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I.  INTRODUCTION 


A.  BACKGROUND 

The  2009  tightening  global,  national,  and  federal  defense  financial  environment 
has  generated  the  requirement  for  all  government  agencies  to  examine  every  possible 
means  to  reduce  the  cost  of  doing  business.  The  possibility  that  there  may  be  overlap  in 
information-technology-related  contracts  between  East  Coast  and  West  Coast  entities 
prompted  Commander,  Naval  Surface  Forces’  (CNSF)  need  for  an  independent  analysis. 
Identification  and/or  modification  of  redundancies  could  yield  cost  savings  while 
improving  efficiencies  in  the  contracting  process.  To  this  end,  three  information 
technology  programs  supporting  CNSF  and  subordinate  units  within  their  claimancy  are 
examined. 

The  purpose  of  this  project  was  to  assess  these  major  IT-related  programs 
providing  services  to  CNSF  staff  personnel  (East  and  West  Coast)  and  subordinate 
commands  within  CNSF’s  purview.  The  programs  of  study  identified  by  the  project 
sponsor,  the  CNSF  Comptroller,  were:  Training  and  Operational  Readiness  Information 
Services  (TORIS),  Continuous  Monitoring  Program  (CMP),  and  the  CNSF  Web.  This 
study  describes  factors  affecting  program  performance  and  evaluation.  The  expectation 
is  to  provide  insights  to  decision  makers  that  can  assist  them  in  determining  the  best 
possible  method  for  achieving  reductions  in  cost,  improvements  in  program  efficiencies, 
and  development  of  performance  standards. 

B.  RESEARCH  QUESTIONS 

The  following  research  questions  apply: 

•  What  programs  are  currently  being  funded? 

•  Why  are  they  being  funded?  What  programs  and  commands  do 
they  affect? 

•  What  commands  do  they  support? 

•  What  are  the  specific  services  to  be  supported  as  identified  by  their 
contract  terms? 

•  Are  there  redundancies  in  services  provided  amongst  the  various 
programs? 
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C.  POTENTIAL  BENEFITS 

The  greatest  potential  benefit  from  this  project  lies  in  making  modifications 
designed  to  cut  costs  and/or  improve  efficiencies  and  reduce  the  possible  redundancies  in 
major  information  technology  programs.  CNSF’s  unique  recent  history  involved 
merging  two  geographically  separated  Type  Commanders  (TYCOMs),  while  allowing 
both  staffs  to  continue  operating  and  maintaining  their  current  level  of  service. 

The  second  benefit  that  this  project  aims  to  provide  is  more  conceptual  in  nature. 
It  aims  to  support  the  strategic  information  superiority  goals  of  the  Department  of  the 
Navy  (DoN).  According  to  the  DoN  Chief  Information  Officer  (CIO): 

One  of  the  underlying  tenets  in  pursuit  of  information  superiority  in  the 
DoN  is  the  ability  to  focus  resources  on  Information  Technology 
investments  that  are  the  most  effective  in  achieving  that  superiority.  This 
is  manifested  directly  by  investing  in  information  technology  that  supports 
the  warfighting  mission  by  providing  secure  information  when  and  where 
it  is  needed.  It  is  also  manifested  by  focusing  on  information  technology 
investments  that  improve  the  mission  and  strategic  objectives  of  all  DON 
organizations,  afloat  and  ashore,  that  directly  or  indirectly  support  the 
warfighting  mission.  (Department  of  the  Navy,  2001a) 

Therefore,  from  the  strategic  perspective,  this  project  aims  to  ensure  that  these 
programs  support  the  warfighter  by  directly  complementing  the  CNSF  mission  and  the 
Surface  Warfare  Enterprise  (SWE)  Charter. 

D.  METHODOLOGY 

The  following  actions  were  taken  to  accomplish  this  project.  First,  the  authors 
met  with  the  project  sponsor  to  determine  the  project  scope  and  breadth,  as  well  as  to 
gather  relevant  background  and  historical  information.  Second,  meetings  were  conducted 
with  the  respective  program  managers  for  the  three  subject  programs.  These  meetings 
were  designed  to  answer  the  who,  what,  where,  when,  and  why  of  each  program,  while 
also  providing  information  on  the  mission,  performance,  management,  financial,  and 
technical  areas  of  each  system.  Finally,  interviews  were  conducted  with  end  users  to 
integrate  their  feedback  into  the  overall  analysis. 
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The  combination  of  the  three  primary  information  sources:  the  project  sponsor, 
the  program  managers,  and  the  end  users,  collectively  forms  the  basis  for  the  research. 

E.  ORGANIZATION  OF  THE  PROJECT 

This  paper  is  separated  into  five  chapters.  The  first  chapter  explains  the  project 
sponsor’s  problem  and  the  approach  used  to  solve  it.  The  second  chapter  provides 
background  information  on  the  project  sponsor’s  command.  In  addition,  it  defines 
baseline  assessment  in  terms  of  information  technology.  The  third  chapter  details  each  of 
the  three  major  programs  identified.  The  fourth  chapter  describes  the  qualities  and 
attributes  of  successful  information  technology  programs,  and  it  presents  the  findings 
from  the  program  manager  and  user  feedback  interviews.  Finally,  the  last  chapter  offers 
the  findings  from  the  research,  including  suggestions  for  follow-on  research. 
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II.  BACKGROUND 


A.  INTRODUCTION 

This  chapter  provides  an  overview  of  a  baseline  assessment  and  its  relationship  to 
information  technology  resources.  In  addition,  it  provides  background  on  the  history, 
organization,  and  staff  objectives  of  Commander,  Naval  Surface  Forces  (CNSF).  The 
goal  of  the  chapter  is  to  establish  a  fundamental  understanding  as  to  what  a  baseline 
assessment  of  CNSF  information  technology  resources  entails,  and  why  an  assessment  is 
important  in  the  subject  situation. 

B.  PROPERTIES  OF  A  BASELINE  ASSESSMENT 

1.  Definition  of  Baseline 

As  a  noun,  a  baseline  is  defined  as  a  starting  point,  one  which  can  be  used  to  draw 
critical  observations  or  data  from  in  the  effort  of  comparison  (baseline,  Merriam-Webster 
Online  Dictionary,  n.d.).  The  implied  intention  would  then  be  to  judge  improvement 
from  the  initial  baseline  to  some  improved  state  (baseline,  BusinessDictionary.com,  n.d.). 

To  convert  baseline  to  an  adjective,  as  done  with  this  project,  it  is  implied  that  we 
are  concerned  with  the  initial  effort  to  draw  observations  and  data  from  the  subject 
system  to  produce  a  standard.  The  goal  of  such  an  exercise  would  then  be  to  identify 
concerns  and  recommended  solutions. 

A  baseline  objective  can  be  obtained  in  various  scenarios.  For  example,  a 
baseline  performance  can  be  derived  from  a  professional  baseball  player’s  rookie  season. 
In  this  case,  numerous  statistical  categories  would  be  assembled  at  the  conclusion  of  the 
season.  While  follow-on  analysis  would  provide  areas  for  improvement,  subsequent 
seasonal  statistics  would  then  be  compared  against  the  baseline  performance  to  measure 
the  success  rate  of  the  analysis. 

Therefore,  baseline  in  this  case  is  the  initial  kind  of  review  to  draw  observations 
and  data  from  the  subject  systems. 
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2. 


Definition  of  Assessment 


An  assessment  is  the  action  of  determining  the  importance,  size,  or  value  of 
something  (assessment,  Merriam- Webster  Online  Dictionary,  n.d.).  In  addition,  it  does 
much  more  than  that.  Simply  assigning  an  assessment  to  be  completed  makes  an 
organization’s  intentions  known  that  the  targeted  program  is  of  significant  importance  to 
leadership  due  to  the  resources  it  promised  toward  its  completion.  Also,  it  dictates  that 
the  organization  must  clearly  spell  out  what  composes,  or  detracts  from,  value  in  the 
target  program.  Finally,  it  implies  that  the  resulting  information  will  be  utilized  in  such  a 
way  as  to  explain  and  improve  performance  (Pittsburg  State  University,  2007). 

In  this  situation,  when  the  project  sponsor  offered  the  original  concept,  all  three  of 
the  above  points  were  made  known  within  the  organization.  Therefore,  while  the  authors 
complete  the  action  of  determining  the  importance,  size,  or  value  of  the  subject  systems, 
the  project  sponsor  made  a  forceful  statement  to  the  stakeholders  of  each  program. 

3.  The  Relationship  between  a  Baseline  Assessment  and  Information 
Technology 

It  is  in  the  best  interest  of  an  organization  to  develop  and  promulgate  strategies, 
goals,  and  missions.  This  enables  the  organization  to  focus  on  what  is  the  best  course  of 
action  and  use  of  resources  to  accomplish  those  goals,  by  way  of  the  assigned  strategy. 
Information  technology  is  an  example  of  one  type  of  resource  that  can  be  used  to  reach  an 
organization’s  goals.  Information  technology  is  not  normally  the  solution,  but  is  instead 
a  means  to  achieve  a  goal.  For  example,  an  organization  may  set  a  goal  to  develop  and 
support  the  families  of  its  personnel.  The  organization  may  then  develop  a  strategy  to 
support  that  goal,  which  may  include  information  technology  systems,  such  as  an  online 
virtual  community  for  information  sharing  and/or  collaboration  amongst  families. 

At  the  time  of  origination,  the  virtual  community  for  information  sharing  may 
have  adequately  supported  the  goal,  as  it  connected  strangers,  linked  by  a  common  bond 
to  each  other  for  grouping  purposes.  However,  as  time  goes  on,  either  the  organization’s 
goal  or  the  characteristics  of  the  virtual  community  may  change  either — perhaps  family 
development  is  no  longer  a  primary  concern  of  management  or  the  operation  of  the 
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system  represents  a  significant  percentage  of  the  discretionary  budget  of  the  organization. 
In  any  case,  it  is  prudent  that  the  organization  review  its  tools  and  methods  it  uses  to 
reach  its  goals. 

Unfortunately,  information  technology  is  unique  in  that  it  involves  hardware  and 
software  systems,  which  are  both  physical  and  logical  in  nature.  This  presents  a 
challenge  for  an  expedient,  periodic  review  because  information  technology,  while 
seemingly  similar,  is  usually  uniquely  configured,  making  it  one-of-a-kind,  technical,  and 
extremely  dynamic. 

So,  a  course  of  action  for  an  organization  to  address  how  it  is  meeting  its  goals  via 
information  technology  is  to  conduct  a  baseline  assessment  of  its  information  technology 
resources.  The  baseline  assessment,  which  is  the  initial  action  to  draw  observations  and 
data  in  order  to  determine  the  importance,  size,  or  value  of  the  subject  systems,  would 
provide  not  only  the  required  guide  for  improvement,  but  also  a  template  for  subsequent 
reviews  in  the  future. 

Finally,  while  federal  law  does  not  mandate  a  review  of  this  type  for  information 
technology  systems  in  operation,  it  is  required  for  major  systems  in  the  acquisition  phase. 
However,  according  to  the  U.S.  Office  of  Management  and  Budget  (OMB),  conducting  a 
baseline  assessment  of  minor  programs  is  good  practice  because  it  provides  the 
organization  with  useful  information  to  compare  already  deployed  and  prospective 
systems  (Government  Accountability  Office,  2009). 

C.  COMMANDER,  NAVAL  SURFACE  FORCES  (CNSF)  BACKGROUND 

1.  History 

Within  the  previous  decade,  the  project  sponsor,  CNSF,  has  undergone  a  period  of 
tremendous  transformation.  The  following  is  a  brief  summary  of  its  recent  history,  as 
found  in  the  Battle  Orders  of  CNSF. 

In  2001,  Commander,  Fleet  Forces  Command  (CFFC)  was  created.  Commander 
in  Chief,  U.S.  Atlantic  Fleet  (CINCLANTFLT),  was  selected  by  the  Chief  of  Naval 
Operations  (CNO)  to  be  concurrent  CFFC,  with  Commander  in  Chief,  U.S.  Pacific  Fleet 
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(CINCPACFLT),  as  the  follow  organization  (in  other  words,  the  second  organization). 
CFFC  was  then  overall  responsible  for  the  integrated  requirements  of  the  manning, 
training,  and  equipping  of  the  Atlantic  and  Pacific  Fleets.  This  is  important  to  note 
because  this  action  was  the  forerunner  for  what  would  occur  later  within  the  more 
specialized  Surface  Fleet. 

Then,  in  2004,  CFFC  established  subordinate  Type  Commander  (TYCOM) 
organizations  for  the  various  warfare  operating  forces.  The  design  installed  a  three-star 
lead  and  a  two-star  follow  TYCOM  on  the  opposite  coast.  At  this  time,  CNSF  was 
designated  as  the  lead  TYCOM  while  acting  concurrently  as  the  Commander  in  Chief, 
U.S.  Pacific  Fleet.  Commander,  Naval  Surface  Force,  U.S.  Atlantic  Fleet  (CNSL)  would 
be  the  follow  TYCOM.  Originally,  CNSF  was  tasked  with  the  development  and 
promulgation  of  policy,  while  CNSL  would  be  responsible  for  the  readiness  of  all 
warships  within  the  claimancy. 

The  following  year,  CNSF  developed  the  Surface  Warfare  Enterprise  (SWE)  in 
accordance  with  the  Navy  Enterprise  Concept;  see  Figure  1.  Its  mission  was  the 
development  of  a  professional  system  of  surface  ships  and  sailors,  able  to  be  employed 
by  the  Combatant  Commanders.  Most  important  to  note  concerning  the  concept  of  the 
SWE  was  its  mandate  to  collaborate  in  advance  with  all  stakeholders,  or  providers,  as 
opposed  to  the  development  within  a  stovepipe.  Example  providers  include  Naval 
Supply  Systems  Command  (NAYSUP)  and  Space  and  Naval  Warfare  Systems  Command 
(SPAWAR). 
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Figure  1.  The  Navy  Enterprise  Concept  (Commander,  Naval  Surface  Forces,  2008b) 

The  CNO  renamed  CFFC  to  Commander,  U.S.  Fleet  Forces  Command 
(CUSFFC)  in  2006,  which  it  is  known  as  to  this  date. 

Also  in  that  year,  SWE  officially  stood  up.  In  addition,  it  formally  designated 
CNSL  as  the  Current  Readiness  Officer  (CRO)  of  the  surface  forces,  and  it  launched 
Class  Squadrons  (CLASSRONs),  based  on  ship  classes,  as  subordinates  to  the  CRO 
(Commander,  Naval  Surface  Forces,  2009). 


2.  Organization  Structure  of  Key  Stakeholders 

The  following  section  describes  the  organization  of  the  Surface  Warfare 
Enterprise,  CNSF,  and  one  staff  code  within  CNSF-the  Comptroller  N00F  code. 

a.  Surface  Warfare  Enterprise 

The  SWE  is  governed  by  the  Surface  Board,  which  is  composed  of  Flag 
Officers  and  Senior  Executive  Service  (SES)  leaders  of  the  various  commands  that 
collectively  compose  the  surface  enterprise.  The  Board  is  headed  by  the  Commander, 
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Naval  Surface  Forces.  The  goal  of  the  Surface  Board  is  to  closely  collaborate  between 
the  various  stakeholders  to  ensure  that  the  maximum  synergy  occurs  as  capabilities  are 
aligned  toward  the  SWE  objectives. 

There  are  eight  Cross  Functional  Teams  (CFT)  that  report  to  the  Surface 
Board  via  the  SWE  Deputy.  These  teams  are  developed  as  multi-disciplinary  from  the 
stakeholder  commands  and  are  mandated  to  focus  on  collaboration  across  each  CFT,  vice 
operation  within  specific  stovepipe  duties  and  responsibilities.  See  Figure  2. 
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Figure  2.  SWE  Organization  Chart  (Commander,  Naval  Surface  Forces,  2009a) 

b.  Commander,  Naval  Surface  Forces 

The  CNSF  organization,  a  three-star  command,  is  considered  one 
organization  with  two  staffs.  While  this  may  seem  redundant,  it  is  important  to 
remember  original  2004  concept  that  CNSF  was  formed  to  fulfill:  one  staff,  CNSP,  was 
designated  as  the  policy  writer  and  the  second,  CNSL,  was  the  supporter  of  warship 
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readiness,  with  the  CLASSRONs  reporting  to  it.  So,  the  two  staffs  do  not  possess 
concurrent  responsibilities,  but  rather  complement  each  other.  They  form  two  halves  that 
combine  to  act  in  unison  as  a  single  staff.  See  Figure  3. 
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Figure  3.  CNSF  Organization  Chart  (Commander,  Naval  Surface  Forces,  2009a) 


c.  CNSF  N00F  Organization 

It  is  important  to  point  out  that  there  is  one  code  within  CNSF  that  does 
not  fit  precisely  into  the  two  staff  construct.  The  Comptroller  department.  Code  N00F,  is 
led  by  a  Department  of  the  Navy  (DoN)  civilian,  located  at  CNSF  headquarters,  who 
manages  two  staffs,  one  on  either  coast.  The  Comptroller  is  responsible  to  CNSF  for  the 
fiscal  accountability  of  the  organization  as  a  whole  and  for  the  proper  distribution  of 
funding.  In  addition,  the  comptroller  submits  future  budget  requests  as  required. 
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Due  to  the  unique  fiduciary  responsibility  of  the  position,  the  Comptroller 
is  the  only  department  head  charged  with  managing  operations  on  both  coasts.  See 
Figure  4. 
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Figure  4.  CNSF  Comptroller  Organization  Chart  (Commander,  Naval  Surface  Forces, 

2007a) 


3.  CNSF  Staff  Objectives 

The  CNSF  staff  objectives  are  developed  in  support  of  the  organization’s  mission 
statement,  which  reads: 

SURFOR  provides  operational  commanders  with  well-trained,  highly 
effective,  and  technologically  superior  surface  ships  and  Sailors.  To 
sustain  peak  levels  of  combat  readiness,  SURFOR  equips  its  forces  with 
the  necessary  training,  tools,  maintenance  and  material  to  successfully 
accomplish  their  mission  —  across  the  entire  spectrum  of  warfare 
operations.  (Commander,  Naval  Surface  Forces,  2009) 
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In  addition,  CNSF  staff  objectives  are  delineated  by  specific  duties  and 
responsibilities,  to  ensure  that  departments  are  properly  aligned  to  avoid  any  duplication 
of  effort.  The  following  section  describes  staff  objectives  by  functional  grouping  in 
greater  detail. 


a.  CNSF/CNSP 

CNSF  is  responsible  as  the  TYCOM  for  the  Pacific  surface  ships  to 
Commander,  U.S.  Pacific  Fleet  for  executing  U.S.  Code  Title  10  responsibilities  for 
manning,  training,  and  equipping  those  ships.  In  addition,  it  answers  to  CUSFFC  for  any 
surface  warfare  request  for  forces.  Also,  CNSP  is  responsible  for  the  management  of  the 
Littoral  Combat  Ship  class.  Finally,  as  the  Pacific  lead,  it  tracks  Pacific  ship  readiness 
via  West  Coast  ISICs  and  CLASSRONs. 

b.  CNSL 

First,  CNSL  is  responsible  as  the  TYCOM  for  the  Atlantic  surface  ships  to 
CUSFFC  for  executing  Title  10  responsibilities  for  manning,  training  and  equipping 
those  ships.  In  addition,  as  CRO,  it  monitors  the  readiness  of  all  surface  ships.  This 
entails  metric  development,  tracking,  and  analysis,  through  the  CLASSRONs,  as  well  as 
collaborating  with  the  SWE  CFTs,  as  required.  Finally,  as  the  Atlantic  lead,  it  tracks 
Atlantic  ship  readiness  via  east  coast  ISICs  and  CLASSRONs. 

c.  Afloat  Training  Groups  (ATG) 

These  organizations,  subordinate  to  the  TYCOMs,  are  tasked  with  the 
training  of  warships  in  the  pre-deployment  phase.  In  addition,  they  conduct  certifications 
that  graduate  surface  ships  to  the  deployment  phase. 

d.  CLASSRONs 

CLASSRONs  are  intermediate  level  organizations  between  the  shipboard 
and  TYCOM  echelons.  They  are  composed  of  analysts  and  support  staff  that  are 
responsible  to  and  work  with  the  CRO  and  its  staff  to  identify  and  resolve  any  issues 
within  the  domain  of  the  surface  fleet’s  readiness. 
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e.  N1 -Manpower  and  Personnel 

The  Manpower  and  Personnel  directorate  splits  its  efforts  between 
TYCOM  and  SWE  responsibilities.  Specifically,  CNSL  N1  is  accountable  for  current 
and  future  manning  and  billeting  readiness  issues  to  the  CRO.  CNSP  N1  leads  the 
Program  Objectives  Memorandum  effort  and  all  civilian  manning  issues. 

/.  A ^-Operations  and  Plans 

The  N3  Department  is  focused  on  the  current  and  future  operations  of  the 
Force.  Both  coasts  detail  deployment  schedules  for  their  ships,  and  then  they  calculate 
dwell  and  home  tempo  times  to  ensure  that  no  limits  are  breached.  CNSL  N3 
additionally  is  the  lead  staff  for  Intelligence  and  Anti-Terrorism/Force  Protection 
concerns.  On  the  West  Coast,  CNSP  N3  manages  hull  swap  and  homeport  shifts  across 
the  fleets. 


g.  N41-Supply  and  Logistics 

In  addition  to  its  TYCOM  responsibilities,  CNSP  N41  is  responsible  for 
all  supply  and  logistics  policy  and  guidance.  CNSL  N41  performs  its  TYCOM 
responsibilities  while  reporting  to  the  CRO  on  all  systemic,  logistics  problems. 

h.  N 6-Combat  Systems  and  C4I 

The  primary  attention  of  CNSP  N6  is  the  future  readiness  of  the  fleet’s 
C5I  capabilities.  CNSL  N6  is  focused  on  the  current  C5I  readiness  of  the  fleet.  In 
addition,  CNSL  N6,  supported  by  CNSP  N6  is  charged  with  the  collection  and 
distribution  of  all  operationally  significant  ships  readiness  data,  for  example,  casualty 
reports  (CASREPs). 


i.  N7 -Training  and  Readiness 

The  most  important  duty  for  CNSP  and  CNSL  N7  is  the  development  of 
training  policy  for  the  fleet.  The  secondary  responsibility  is  monitoring  the  training 
readiness  of  the  fleet.  Similar  to  the  other  codes,  CNSP  is  designated  as  the  lead  for  this 
effort,  while  CNSL  is  the  follow  staff. 
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N 8-Force  Requirements  and  Assessments 


N8  is  a  staff  code  focused  on  developing  policy  to  support  the  future 
requirements  of  the  fleet.  Due  to  its  close  relationship  with  CUSFFC,  the  lead  staff  is 
located  at  CNSL  headquarters  vice  CNSP.  The  functions  of  the  Department  are  varied, 
but  mostly  cover  the  following  areas:  coordination  with  the  SWE  Strategic  Financial 
Management  CFT  to  lead  the  Force’s  Planning,  Programming,  Budgeting,  and  Execution 
efforts,  development,  and  tracking  of  metrics  for  both  individual,  unit,  and  force  level 
requirements,  and  management  of  the  organization’s  Joint  Capability  Integration  and 
Development  System  (Commander,  Naval  Surface  Forces,  2009a). 
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III.  CNSF  IT  SYSTEMS  REVIEW  AND  DESCRIPTION 


To  quote  Thomas  Friedman,  “the  world  is  flat,”  and  as  Figure  5  shows,  the  same 
is  true  for  the  major  information  systems  we  profiled  at  CNSF  (Friedman,  2005).  While 
they  are  considered  internal  programs  for  management  sake,  all  three  are  connected  with 
external  stakeholders  in  some  form  or  fashion.  For  example,  CNSF  Web  provides 
information  to  families,  media,  and  researchers,  while  Training  and  Operational 
Readiness  Information  Services  (TORIS)  and  Continuous  Monitoring  Program  (CMP) 
communicate  through  the  Navy  Readiness  Reporting  Enterprise  (NRRE)  to  the  Pentagon 
to  provide  real-time  status  of  forces  updates.  As  Figure  5  depicts,  both  CMP  and  TORIS 
feed  the  Defense  Readiness  Reporting  System  (DRRS)  via  its  Navy  component  system. 


17 


GFM 

ORG 


ORG 


NAVY  GFM 
ORG 


cnic  pccc 

LJ 


NTIMS 


( 


DRRS 


CRl"  rfkia  *if:i:-TKri  l(: 
KCb,  IHtfS.  Nl  h'a  and 
nRRft-N 


SUB  SURFACE  AR 


NML I  and  KbADINfcSS- 


DRRS-N 


httpsiV/dnsn  flc.na  vy .  smil.  mil/DRRSN 


□  RG  SORTS  .a-vl 
Lcnacv 


2K's 

NDE 

WEBSKED 

CDMD-OA 

CASREPS 

IPARS 

PMS  SKED 

R-SUPFLY 

EDFS 

AWN 

NMD 

R-  ADMIN 

SOWS 

VSB 

CQA  Toolkit 

RMMCG 

RCE 


TRMS 


Figure  5.  Relational  Overview  between  TORIS,  CMP,  and  DRRS  (Commander,  Naval 

Surface  Forces,  2008c) 


This  chapter  contains  an  in-depth  discussion  of  each  of  the  three  major 
information  systems:  TORIS,  CMP,  and  CNSF  Web.  It  expressly  details  their 
background,  mission  needs,  and  technical  requirements.  In  addition,  it  provides  a  brief 
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system  description  for  each.  The  intent  of  the  section  is  to  provide  the  user  with  a  basic 
understanding  of  what  each  system  is,  how  it  works  the  way  that  it  does,  and  what 
mission  it  is  designed  to  support. 

A.  TRAINING  AND  OPERATIONAL  READINESS  INFORMATION 

SERVICES  (TORIS) 

TORIS  is  a  Web-centric  data-engine  that  receives  inputs  from  afloat  and  ashore 
units.  It  stores,  displays,  and  transmits  the  data  to  external  information  systems  upon 
request  or  via  automatic  schemes.  Its  focus  is  the  training  readiness  of  the  surface  force. 

1.  System  Background 

TORIS  was  developed  immediately  following  the  terrorist  attacks  of  9/11.  At  the 
time,  a  requirement  was  identified  to  provide  an  instant  snapshot  of  the  current  training 
readiness  of  the  surface  fleet  to  leadership.  The  intent  of  the  program,  the  development 
task  of  which  was  delegated  to  mobilized  reservists,  was  to  enable  the  Immediate 
Superior  in  Command  (ISIC)  and  Type  Commanders  (TYCOMs)  with  the  ability  to 
identify  warships  ready  for  deployment  to  the  Combatant  Commanders.  It  did  so  by 
providing  Commanding  Officers  at  the  unit  level  with  a  current  view  of  their  ship’s 
training  readiness,  which  could  be  compared  to  the  ATG’s  required  metrics  (Commander, 
Naval  Surface  Forces,  2008d). 

In  2003,  the  Chief  of  Naval  Operations  (CNO)  directed  Commander,  U.S.  Fleet 
Forces  Command  (CUSFFC)  to  develop  a  plan  that  would  substantially  improve  the 
Navy’s  deployed  warfighting  capability.  This  was  in  response  to  concerns  that  only  a 
small  percentage  of  Navy  vessels  were  deployed  or,  worse  yet,  deployable  at  any  one 
point  in  time.  Also,  there  were  specific  concerns  that  the  ISICs  and  TYCOMs  did  not 
have  a  firm  grasp  on  the  readiness  of  units  still  in  the  pre-deployment  training  phases. 
So,  CUSFFC  promulgated  the  Fleet  Response  Plan  (FRP),  which  the  CNO  approved  in 
May  2003.  The  plan  required  that  the  Navy  be  able  to  deploy  eight  carrier  strike  groups 
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within  90  days  of  notification,  and  it  was  known  as  the  “6  +  2”  goal.  To  do  so  would 
require  a  massive  improvement  in  the  Navy’s  training  cycle  (Government  Accountability 
Office,  2005). 

So,  in  March  2004,  CNSF  introduced  a  new  training  process  called  SHIPTRAIN. 
According  to  CNSF: 

SHIPTRAIN  is  the  primary  means  to  align  and  integrate  processes  to 
effectively  and  efficiently  produce  warships  and  trained  crews 
continuously  ready  for  operational  mission  tasking,  which  is  the  product 
of  the  Surface  Warfare  Enterprise  (SWE).  (Commander,  Naval  Surface 
Forces,  2007b). 

It  was  developed  to  support  “Sea  Power  21,  the  training  requirements  of  the  Navy 
Mission  Essential  Task  List  (NMETL),  and  the  Navy  Warfare  Training  System  (NWTS)” 
(Taylor,  2004).  To  do  so,  it  shifted  unit  level  training  to  a  continuous  process,  thus 
reducing  the  time  requirement  for  training  and  follow-on  assessment  and  certification  by 
75%  (SURFOR  Public  Affairs,  2006).  As  Vice  Admiral  Etnyre,  the  former  Commander, 
Naval  Surface  Forces,  wrote,  “Gone  are  the  days  of  ‘ramping  up’  to  deploy.  We  have  to 
be  ready  when  called.  Training  MUST  prepare  us  to  fight.  Continuous  training  will  do 
that  and  keep  us  ready-all  the  time,  anywhere”  (Etnyre,  2007). 

The  primary  tools  that  enabled  SHIPTRAIN’s  efficiencies  were  TORIS  and 
Training  Figure  of  Merit  (TFOM).  TFOM  was  a  software  application  that  tracked  ship’s 
training  data  in  various  warfare  categories.  It  initially  tracked  only  Engineering,  Strike 
and  Air  warfare  areas.  However,  over  the  following  two  years,  it  added  functionality  to 
support  all  warfare  areas.  It  achieved  its  efficiencies  by  eliminating  duplicative  training 
requirements  and  populating  its  results  on  the  ship’s  local  area  network  for  command 
leadership  review  and  action. 

In  2005,  TORIS  absorbed  TFOM  as  one  of  its  primary  applications.  From  then 
on,  units  could  populate  the  TFOM  application  via  Web-centric  TORIS,  which  stored  the 
data  and  allowed  authorized  stakeholders  to  view  and  compare  data  to  assess  current 
training  readiness.  While  enhancing  visibility  off  hull,  it  also  was  designed  to  be  user- 
friendly  (Irwin,  2005). 
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In  2006,  Afloat  Training  Group,  U.S.  Pacific  Fleet  (ATGPAC),  the  responsible 
command  for  TORIS,  received  an  award  from  CIO  Magazine.  The  command  was 
recognized  for  its  successful  design  and  implementation  of  TORIS.  According  to  the 
Navy  press  release,  TORIS  was  awarded  due  to  its  “reduction  of  administrative  overhead 
that  was  previously  needed  to  store,  compile,  organize  and  extract  data,  making  training 
more  efficient  by  providing  a  Web-based  tool  for  inputting  data  and  transmitting  it  to  a 
central  data  warehouse  for  analysis”  (Ludwick,  2006). 

2.  Mission  Need 

TORIS  supports  the  training  readiness  of  the  fleet.  By  doing  so,  it  builds  toward 
the  currently  available  war-fighting  capability  of  the  force.  It  provides  for  these  goals  via 
three  major  routes.  The  first  deals  with  the  onboard  training  record  processes.  The 
second  is  through  the  time-shortened  inspection  certification  processes.  The  third  is  via 
the  automatic  population  of  DRRS  via  TORIS’s  TFOM  metrics. 

First,  the  onboard  training  record  processes  provide  a  unit’s  command  leadership 
with  an  instant  look  at  their  current  training  proficiency  within  a  multitude  of  areas.  This 
allows  onboard  management  to  develop  plans  to  improve  weaker  areas  and  reinforce 
stronger  areas,  as  required,  in  the  most  efficient  manner.  This  efficiency  is  achieved 
because  plans  can  be  drawn  that  focus  on  the  weakest  areas  first,  to  ensure  that  the  crew’s 
most  precious  resource — time — is  not  wasted  on  activities  that  will  not  provide  as  great  a 
marginal  benefit. 

Second,  the  inspection  certification  processes  are  now  shortened  because  of  the 
applications  that  TORIS  handles.  Shortening  the  time  spent  on  inspections  is  critical  to 
getting  ships  approved  for  deployment  quicker,  which  assists  the  TYCOMs’  efforts  to 
meet  the  6  +  2  goal. 

Finally,  the  automatic  population  of  DRRS-N  via  TORIS’s  TFOM  metrics  is  the 
last  action  that  TORIS  takes  to  support  the  war-fighting  capability  of  the  force: 
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DRRS-N  provides  the  unique  ability  to  ascertain  and  report  to  the  Office 
of  the  Secretary  of  Defense  (OSD)  the  readiness  level  of  any  Navy  unit,  so 
that  critical  decisions  can  be  made  to  deploy  units  in  a  timely  manner 
based  on  accurate,  up-to-date  information.  (Innova  Systems  International, 

2009) 

TORIS,  therefore,  populates  the  training  data  required  by  DRRS-N  so  that 
TYCOMs  and  OSD  receive  current  data  on  the  training  readiness  of  the  fleet. 

3.  Technical  Requirements 

While  TORIS  is  a  hardware  and  software  information  technology  system,  it  is 
software  intensive.  This  is  because  it  is  essentially  a  suite  of  software  systems,  bundled 
together  in  one  user  interface,  managed  by  four  hardware  servers.  The  complexity  of  the 
system  grows  as  the  interaction  between  the  software  systems  increase,  thus  forcing 
TORIS  system  developers  to  actively  perform  design  functions  that  support  system 
agility. 

The  system  begins  with  the  end  users,  who  populate  their  ship’s  training  data  or 
inspection  records  on  laptop  computers,  tablet  personal  computers,  or  the  ship’s  LAN 
computers.  The  laptops  and  tablets  are  preinstalled  with  remote  software,  so  that 
individuals  can  record  data  without  Internet  connectivity,  if  required.  Later,  individuals 
connect  to  https://toris.atgpac.nmci.navy.mil/wrapper/default.aspx  and  upload  their  data. 
Any  standard  browser  can  be  used  to  access,  but  users  must  authenticate  via  public  key 
infrastructure  (PKI).  The  system  is  programmed  to  operate  within  IT-21  (Afloat)  and 
NMCI  (Shore)  environments. 

The  data  flows  via  unclassified  but  128-bit  secure  socket  layer  to  the  system 
server  array,  see  Figure  6.  Two  application  and  two  database  servers  compose  the  server 
array.  All  four  are  located  onboard  Naval  Base  Coronado,  Naval  Air  Station  North  Island 
installation,  Grace  Hopper  Building.  The  application  servers  utilize  ASP.Net 
programming.  The  database  servers  are  managed  by  Microsoft  SQL  Server  2000/2005 
software. 

Upon  collection  and  storage  of  the  data,  the  system  can  then  respond  to  user 
requests  for  information.  One  type  of  user  request  is  Web-based,  and  it  allows  authorized 
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users  to  investigate  specific  analysis  tools,  such  as  TFOM,  to  gauge  training  readiness 
and  conduct  planning  for  follow-on  activities.  The  second  type  of  request  is  an 
extensible  markup  language  (XML)  query  from  other  information  systems  and 
commands.  This  data  flow  of  various  metrics  and  reports  is  automated,  and  it  feeds 
systems  such  as  the  Navy  Training  Information  Management  System,  (NTIMS),  DRRS- 
N,  and  required  reports  for  the  CLASSRONs.  TORIS  is  able  to  conduct  these  processes 
because  it  “is  the  authoritative  data  source  for  crew  proficiency  and  certification  data  for 
all  surface  forces”  (Commander,  Naval  Surface  Forces,  2008d).  Its  Department  of  the 
Navy  Application  and  Database  Management  System  (DADMS)  information  is  current, 
which  is  important  as  DADMS  is  the  official  DoN  record  for  all  authoritative  data 
sources  throughout  the  Department.  Additionally,  TORIS  is  in  possession  of  a  current 
Authority  to  Operate  (ATO).  Finally,  since  it  contains  Personally  Identifiable 
Information  (PII),  system  administrators  maintain  sufficient  Privacy  Act  requirements. 


Figure  6.  TORIS  System  Architecture  (Commander,  Naval  Surface  Forces,  2008d) 
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4.  System  Description 

As  CNSF  and  the  ATGs  updated  the  Surface  Force  Training  Manual  (SFTM), 
TORIS  system  developers  responded  by  increasing  functionality  to  the  system.  Current 
applications,  as  depicted  in  Figure  7,  included  within  the  TORIS  enterprise  system 
include: 

•  TORIS-Core:  The  central  data  collection  application  used  by 
ATGs. 

•  TORIS-Afloat:  The  data  collection  application  used  by  ship’s 
company  and  ISICs.  This  system  is  deployed  throughout  the 
Surface  Fleet. 

•  Training  Figure  of  Merit  (TFOM):  The  Commanding  Officer’s 
tool  utilized  to  monitor  training  readiness.  This  system  is  a  lens 
deployed  within  TORIS-Afloat  and  is  a  view  within  the  Fleet 
Views. 

•  Fleet  Views:  A  consolidated  view  of  all  data  returned  from  the 
fleet  that  includes  TFOM  views  for  each  afloat  unit. 

•  Unit  Level  Training  Status  (Stoplights):  CNSF’s  authoritative  data 
source  for  ship  certifications  status  across  all  mission  areas 
assigned  by  the  SFTM. 

•  Strike  Group  Matrix  (SGM):  The  authoritative  data  source  for  ship 
certification  dates  across  all  mission  areas  assigned  by  the  SFTM. 

•  Master  Scheduling  Program  (MSP):  ATG’s  tool  for  scheduling 
training  missions  and  assigned  personnel,  collection  and  display  of 
manpower  expenditure  data. 

•  Ships  in  Training  (SIT):  ATG’s  tool  for  monitoring  ship’s  Unit 
Level  Phase  training  progress  towards  obtaining  or  maintaining 
certification  during  ULTRA-C,  ULTRA-E,  and  ULTRA-S  phases 
of  the  SHIPTRAIN  cycle. 

•  TORIS-Supply:  ATG’s  tool  to  conduct  Supply  Management 
Certifications  per  CNSFINST  5040.1. 

•  TORIS-3M:  ATG’s  tool  to  conduct  3-M  Certifications  per 
CNSFINST  4790.13. 

•  ATG  Enterprise  Personnel  Program  (AEPP):  ATG’s  Human 
Resource  management  tool.  (Commander,  Naval  Surface  Forces, 
2009b) 
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Figure  7.  TORIS  Portal  Screenshot  (Commander,  Naval  Surface  Forces,  2008d) 


The  data  flow  of  the  system  is  extremely  important,  considering  TORIS  is 
essentially  a  data-engine — an  electronic  system  of  hardware  and  software  that  receives 
data  as  an  input  and  conducts  specified  tasks  on  the  inputs.  The  tasks  can  either  be  on- 
demand  by  users,  such  as  stoplight  views  for  ship  Commanding  Officers,  or  pre¬ 
programmed  by  system  administrators,  such  as  DRRS-N  TFOM  reporting.  Figure  8 
explains  the  primary  data  requests,  flows,  applications,  and  databases  utilized  by 
stakeholders. 
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TORIS  -  Data  Architecture 


Figure  8. 


TORIS  Data  Architecture  (Commander,  Naval  Surface  Forces,  2008d) 


There  are  over  10,000  authorized  users  within  the  system,  of  which  over  4,000  of 
them  were  active  within  the  past  90  days  in  August  2009.  Shipboard  users  account  for 
3,340  of  the  total  authorized  users,  representing  167  different  units  with  approximately 
twenty  users  per  vessel. 

According  to  the  TORIS  system  managers,  the  following  are  the  primary, 
secondary,  and  tertiary  areas  of  interest  (AOI)  within  TORIS  by  echelon. 
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Echelon 

Primary  AOI 

Secondary  AOI 

Tertiary  AOI 

Fleet 

TORIS-Afloat 

Stoplights 

Fleet  Views 

ATG 

Stoplights 

Ships  in  Training 

MSP 

CLASSRON 

Fleet  Views 

Metrics 

Reports 

ISIC 

Stoplights 

Ships  in  Training 

Fleet  Views 

Table  1.  Most  Popular  TORIS  Applications  by  Echelon  (From  Stegner,  2009) 


a.  Contract  Specifications 

In  2009,  the  system  costs  were  $2,303,220.  The  contract  statement  of 
work  spells  out  the  requirement  for  seven  key  functional  areas: 

•  Functional  Business  Analysis:  to  provide  process  improvement  and 
return  on  investment  solutions  for  ATG 

•  Integration:  to  provide  the  software  design  and  engineering 

required  to  support  the  interoperability  of  TORIS  with  external 
information  systems 

•  Configuration  Management/Quality  Control:  to  provide  the  testing 
and  assurance  that  users’  needs  are  met  and  DoD  information 
technology  requirements  are  satisfied 

•  Database  Management/System  Administration:  to  develop  the 

database  while  managing  the  authorized  user  list 

•  User  Training  Services:  to  train  afloat  and  ATG  users  on  and 
develops  training  documentation  on  all  system  changes 

•  Fleet  Hardware  Procurement/Distribution  and  Software 

Installation:  to  manage  the  hardware  and  software  install  at  ATG 
and  afloat  locations 

•  Help  Desk  Management  and  Documentation:  to  run  the  TORIS 
help  desk  to  field  trouble  calls 

The  statement  of  work  also  calls  for  the  following  twenty  personnel 
positions  to  be  provided  by  the  contractor: 

•  One  Program  Manager/Chief  Engineer 

•  Two  Senior  Software  Engineers 
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•  Three  Mid-Grade  Software  Engineers 

•  Three  Junior  Software  Engineers/Hardware  Technicians 

•  One  Senior  Quality  Assurance  Engineer/Systems  Analyst 

•  One  part-time  Senior  Accreditation  Specialist 

•  One  Database  Architect/Data  Quality  Engineer 

•  One  Technical  Documentation  Specialist/Quality  Assurance  Test 
Engineer 

•  One  Junior  Subject  Matter  Expert/Data/Systems  Analyst 

•  Three  Mid-Grade  Subject  Matter  Experts/Data/Systems  Analysts 

•  One  Senior  Subject  Matter  Experts/Data/Systems  Analyst 

•  Two  SWE  Senior  Grade  Subject  Matter  Experts/Data/System 
Analysts  (Fleet  and  Industrial  Supply  Center — San  Diego,  2008) 

5.  Summary 

TORIS,  a  $2.3  million  per  year  program,  directly  supports  Fleet  training 
readiness.  It  does  this  by  transmitting  and  displaying  key  training  data  to  commanders 
afloat  and  planners  ashore  to  improve  the  surface  force’s  ability  to  meet  the  readiness 
demands  of  the  Navy’s  Fleet  Response  Plan. 

B.  CONTINUOUS  MONITORING  PROGRAM  (CMP) 

CMP  is  a  software  and  hardware  system  that  transmits  precise  and  current  supply 
and  financial  management  data  to  and  from  ships,  submarines  under  the  purview  of 
CNSF,  and  shore  establishments.  Its  goal  is  to  improve  readiness  while  reducing  man¬ 
hours  by  automating  key  data  draw-downs  and  tedious  manual  data  entry. 

1.  System  Background 

As  originally  conceived  in  1998,  CMP  was  planned  as  an  information  system  that 
would  provide  two  solutions  to  the  Fleet.  First,  it  was  designed  to  act  as  a  management 
tool  for  shipboard  Supply  Department  personnel  to  quickly  identify  specific  discrepancies 
within  the  Stores  Division  (SI)  and  the  Food  Service  Division  (S2).  It  was  hoped  that 
follow-on  investigation  by  ships’  crews  would  not  only  correct  the  discrepancies,  but  also 
would  lead  management  to  discover  the  origin  of  the  discrepancy.  Upon  discovery  of  the 

28 


root  cause,  Supply  Department  leadership  could  repair  faulty  shipboard  procedures  and 
conduct  workplace  training  in  order  to  reduce  the  chances  of  future  failures. 

Additionally,  off  hull,  CMP  was  designed  to  provide  a  quick  and  effective  means 
of  evaluation  of  the  current  Supply  Department  readiness  of  individual  hulls,  squadrons, 
and  classes.  The  CMP  data  submission  allowed  supervisors  on  shore  remote  access  to 
the  readiness  and  overall  health  of  the  Supply  Department.  It  also  could  be  run  by 
onboard  inspectors  and  support  teams  as  a  means  of  either  conducting  or  preparing  for  an 
official  naval  inspection. 

The  system  was  initially  drawn  up  by  CINCLANTFLT  staff  in  1998  for 
installation  on  all  Atlantic  Fleet  surface  ships  and  submarines.  Those  ships  received  the 
program  in  1999  and  2000.  Following  in  the  East  Coast’s  footsteps,  the  Pacific  Fleet 
mandated  its  use  for  West  Coast  surface  ships  and  submarines.  In  2001,  the  installation 
of  CMP  was  complete  for  the  Pacific  Fleet. 

In  2003,  the  program  was  improved  to  include  additional  financial  reporting  data 
from  various  Supply  Department  Divisions,  in  order  to  provide  a  more  complete  snapshot 
of  the  financial  status  of  the  fleet  to  the  TYCOM  comptrollers  and  N41  staffs.  In 
particular,  financial  reporting  was  enhanced  to  provide  specific  Budget  Operating  Target 
Report  (BOR)  data  from  SI  Division,  port  cost  information,  some  Disbursing  (S-4)  data, 
additional  food  service  information,  fuel  data,  and  information  from  Retail  Sales  division 
(S-3)  would  be  captured  (Commander,  Naval  Surface  Forces,  2008a). 

2.  Mission  Need 

The  primary  mission  need  of  CMP  is  fleet  readiness.  It  supports  this  goal  because 
of  the  way  that  it  is  designed  to  bring  attention  to  potential  supply  department 
deficiencies,  which  could  degrade  into  significant  problem  areas  for  ships  in  training  or 
on  deployment.  CMP  is  designed  to  support  this  goal  in  the  following  approach.  First, 
the  N41  code  of  both  the  surface  and  submarine  TYCOMs  assess  their  fleets.  They 
develop  shipboard  supply  department  policy  that  is  designed  to  set  individual  ships’ 
supply  departments  up  for  success,  and  thus  improve  ship  readiness. 
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For  example,  once  a  piece  of  equipment  fails  onboard  a  ship,  troubleshooting 
occurs  to  determine  the  source  of  failure  and  the  best  method  for  repair.  If  a  replacement 
part  is  required,  the  supply  department  will  be  contacted  for  action.  The  part  can  either 
be  located  onboard  in  ship’s  stock  or  stored  off  hull  at  a  repair  part  warehouse.  If  the  part 
is  not  onboard,  a  direct  turnover  requisition  must  be  generated  and  routed  via  military 
standard  requisitioning  and  issue  procedures  through  the  supply  system  for  sourcing. 
Once  sourcing  is  determined,  the  piece  part  is  shipped  to  the  unit’s  designated  location. 
At  that  time,  supply  personnel  receive  the  part  and  immediately  provide  it  to  the 
respective  division’s  repair  parts  petty  officer. 

Unfortunately,  mistakes  lead  requisitions  to  delay  in  suspense  indefinitely  until 
discovered  and  restored  by  ship’s  force.  Theoretically,  a  suspended  requisition  could  be 
delayed  ad  infinitum,  thereby  preventing  the  original  broken  piece  of  equipment  from 
ever  being  repaired. 

Since  this  set  of  procedures  can  occur  multiple  times  in  a  single  day  onboard 
afloat  units,  and  since  any  number  of  requisitions  may  be  in  suspense  at  any  one  time, 
TYCOM  N41  staffs  developed  specific  policies  regarding  the  validity  of  shipboard  parts 
requisitions.  If  the  percentage  of  invalid  supply  department  requisitions  increases  above 
the  TYCOM  standard,  then  the  SI  division  is  in  violation  of  established  force  policies, 
and  it  would  fail  the  applicable  inspection  criteria.  CMP,  which  possesses  the  ability  to 
scan  all  SI  records  in  a  matter  of  minutes  identifies  and  counts  the  number  of  invalid 
requisitions,  can  therefore  improve  ship’s  readiness  by  enforcing  policies  through 
minimal  intervention  by  allowing  drill-down  capability  of  those  specific  pulse-point 
violations. 

The  efficient  and  effective  nature  of  the  system  has  the  added  benefit  of  reducing 
man-hours  spent,  either  onboard  or  on  shore,  analyzing  ship’s  current  data  for  trouble 
spots.  The  automation  process  provides  data  in  minutes  as  opposed  to  two  weeks,  as 
estimated  by  system  developers  (Commander,  Naval  Surface  Forces,  2008a). 

CMP  supports  fleet  readiness  by  drawing  down  data,  called  pulse  points,  from 
other  supply  department  information  systems,  such  as  R-Supply,  Micro-Snap,  Food 
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Service  Management  (FSM),  and  ROM-II,  and  comparing  that  data  against  Fleet 
standards  to  highlight  problematic  areas  before  significant  issues  can  arise. 

The  total  list  of  CMP  S-l  pulse  points  are  as  follows: 

•  Automatic  Reorder  Restriction  Code  (ARRC):  Displays  stock 
records  containing  an  ARRC  and/or  a  limit  flag.  These  can  be 
reviewed  for  accuracy. 

•  AT6  Count:  Number  of  excess  line  items  onboard. 

•  Average  AT6  Dollar  Amount:  Average  dollar  value  of  excess 
material  onboard. 

•  Depot  Level  Repairable  (DLR)  Carcass  Charges:  Current  fiscal 
year  DLR  carcass  charges,  including  surveys,  expressed  as  a 
percentage  of  DLR  obligations. 

•  DLR,  Prior  Fiscal  Year:  Prior  fiscal  year  DLR  carcass  charges, 
including  surveys,  expressed  as  a  percentage  of  DLR  obligations. 

•  Stock  Record  File  Maintenance:  The  number  of  stock  records 
having  invalid  data,  e.g.,  on  hand  balance  with  no  location. 

•  Gross  Effectiveness:  For  current  month  and  a  four-month  average. 

•  Net  Effectiveness:  For  current  month  and  a  four-month  average. 

•  Range:  Percent  of  allowed  items  with  at  least  one  on  hand. 

•  Depth:  Percent  of  allowed  items  having  on  hand  equal  to  or  greater 
than  the  established  requisitioning  objective. 

•  Material  Outstanding  File  (MOF)  Maintenance:  Percent  of 
requisitions  in  the  MOF  with  valid  status. 

•  Internal  Material  Obligation  Validation:  Percent  of  requisitions  in 
the  MOF  having  an  open  Job  Sequence  Number. 

•  Reorder  Review  Value:  The  dollar  value  of  stock  items  not  on 
hand  and  not  on  order. 

•  Reorder  Review  Count:  The  number  of  stock  items  not  on  hand 
and  not  on  order. 

•  Requirement  Processing:  The  number  of  requirements  over  15 
days  old  awaiting  department  head  approval. 

•  Selected  Item  Management  (SIM)  Zero  Balance:  The  number  of 
SIM  items  having  zero  on  hand  balance. 
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The  total  list  of  CMP  S-2  pulse  points  are  as  follows: 

•  FSM  Inventory  Accuracy:  The  most  recent  inventory  percentage 
reported  within  the  FSM  program. 

•  Food  Service  Daily  Posting:  The  total  number  of  days  that  the 
recordskeeper  updated  information  within  FSM. 

•  FSM  Total  Inventories  Conducted:  The  total  number  of  inventories 
conducted  during  the  current  month. 

•  Over/Under  Issue  for  the  Month:  The  dollar  value  measuring 
whether  the  mess  is  on  target  for  its  yearly  allocation. 

•  Suppo/Food  Service  Officer  (FSO)  Audits:  The  total  number  of 
times  the  FSO  conducted  audits  in  FSM  for  the  month. 

The  total  list  of  CMP  S-3  pulse  points  are  as  follows: 

•  Stock  Turn:  The  quotient  between  the  ship’s  store  cost  of  sales 
dollar  value  and  the  current  quarters  opening  inventory  value. 

•  Financial  Difference:  The  quotient  between  the  net  gains  and 
losses  from  all  sales  outlets  and  bulk  storerooms  and  the  total  value 
of  sales,  expressed  as  a  percentage. 

•  Retail  Gross  Profit:  The  result  of  taking  100% — 100(cost  of  sales 
for  the  retail  outlets  and  snack  vending  /  total  retail  sales  for  the 
same). 

•  Canned  Gross  Profit:  The  result  of  taking  100% — 100(cost  of  sales 
for  the  canned  vending  machines  /  total  retail  sales  for  the  same). 

•  Emblematic  Percentage:  The  quotient  between  the  dollar  value  of 
emblematic  inventory  and  the  total  inventory  dollar  value. 
(Commander,  Naval  Surface  Forces,  n.d.) 

3.  Technical  Requirements 

All  CMP  equipment,  data,  and  software  is  government  owned  and  contractor 
operated.  The  CMP  system  is  composed  of  two  primary  components.  The  first  is  a 
software  program  installed  in  shipboard  supply  department  computers  called  the 
extractor.  It  is  written  in  Visual  Basic  and  formatted  to  operate  on  Windows  98,  2000, 
and  XP  systems.  The  program  contains  a  set  of  structured  query  language  (SQL)  queries, 
which,  when  run,  automatically  extract  the  required  data  from  the  applicable  supply 
information  systems.  See  Figure  9. 
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Figure  9.  CMP  S-l  Application  (Commander,  Naval  Surface  Forces,  2008a) 

Personnel  may  conduct  any  of  three  events  after  program  completion.  First,  they 
may  display  the  results  in  Microsoft  Excel  format  for  easy  sorting  and  viewing  of 
individual  records.  See  Figure  10.  This  feature  is  designed  to  provide  shipboard  supply 
departments  with  the  precise  information  they  need  for  the  investigation  and  resolution  of 
discrepancies. 
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ON  BOARD 
CMP  EXTRACTOR 


Figure  10.  CMP  S-l  Pulse  Point  Data  Drill  Down  (Commander,  Naval  Surface  Forces, 

2008a) 

Second,  users  may  print  out  a  management  report,  which  provides  traffic  light 
results,  of  green  or  acceptable,  yellow  or  warning,  and  red  for  unsatisfactory.  See  Figures 
11,  12,  and  13  for  S-l,  S-2,  and  S-3,  respectively.  Primarily,  this  function  is  to  aid 
command  and  squadron  leadership  to  quickly  survey  the  health  of  the  department. 
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Figure  11.  CMP  S-l  Pulse  Point  Traffic  Light  Report  (Commander,  Naval  Surface 

Forces,  2008a) 
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Figure  12.  CMP  S-2  Pulse  Point  Traffic  Light  Report  (Commander,  Naval  Surface 

Forces) 
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Figure  13.  CMP  S-3  Pulse  Point  Traffic  Light  Report  (Commander,  Naval  Surface 

Forces) 


Third,  users  are  required  at  prescribed  periodicities  to  save  the  data  to  file  for 
ultimate  submission  via  unclassified  e-mail  to  the  CMP  server — the  second  major 
component  of  the  CMP  system,  which  receives,  sorts,  stores,  and  presents  the  shipboard 
data  to  authorized  users  at  https://cmp.surfor.navy.mil.  The  server  runs  Microsoft 
Windows  Server  2003.  It  stores  required  data  via  Microsoft  SQL  Server  2003  database 
software.  It  is  composed  of  two  parts:  the  data  server,  which  is  actually  the  CNSF  Web 
Server  located  in  Norfolk,  Virginia,  and  the  Web  server,  physically  located  within  CNSL 
N6  spaces.  The  server  uses  several  programming  languages  (e.g.,  ASP,  XML,  and  PHP), 
to  present  all  the  results  to  the  viewer  within  a  Web  interface.  It  operates  within 
Navy/Marine  Corps  Intranet  underneath  the  CNSF  domain. 
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In  addition  to  providing  users,  who  are  primarily  from  the  TYCOM,  CLASSRON, 
and  ATG  staffs,  the  server  also  populates  several  other  information  systems,  not  the  least 
of  which  is  DRRS-N.  See  Figure  14.  CMP  provides  95%  of  the  required  Supply  Figure 
of  Merit  (SFOM)  data  to  DRRS-N. 
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Figure  14.  CMP  Data  Flow  (Commander,  Naval  Surface  Forces,  2008a) 

Behind  the  scenes  of  CMP  is  the  development  server.  It  is  maintained  by  the 
development  team  of  contractors  and  operates  on  a  dedicated  server,  which  mirrors  the 
Production  Environment,  while  allowing  for  modification,  testing,  and  evaluation. 

4.  System  Description 

CMP  is  managed  through  a  continuing  contract.  The  statement  of  work  explicitly 
calls  for  the  contractor  to: 

Maintain  and  develop  requirements  for  SURFOR  and  SUBFOR  Web 
Development  and  CMP  working  with  staff  personnel.  Continue  current 
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development  and  programming  efforts  for  CMP  and  perform 
administrative  functions  to  track  ongoing  progress.  Assist  Fleet  Units  and 
Subordinate  Commands  in  their  Web  Development  and  CMP’s  expansion 
to  all  fleet  units.  (Fleet  and  Industrial  Supply  Center — Norfolk,  2008) 

Essentially,  contractor  tasks  are  devoted  to  two  primary  tasks.  First,  to  run  the 
engine  of  CMP,  as  is.  This  requires  the  contractor  to  administer  the  current  Web  server 
in  accordance  with  customer  demands,  to  troubleshoot  and  maintain  both  the  Web  server 
and  the  shipboard  extractor,  to  provide  training  to  end  users,  and  to  collect  and 
disseminate  desired  management  data. 

Second,  the  contractor  is  responsible  for  future  development  of  the  system  to  meet 
emerging  requirements.  This  is  a  prominent  function  of  the  system  managers  considering 
that  several  major  functions  have  been  added  to  the  original  template  and  several  more 
are  currently  under  development.  For  example,  new  production  tools  allow  for  TYCOM 
comptrollers  to  grant,  augment,  and  advance  units’  Operating  Targets  (OPTARs)  or 
ship’s  budgets.  Also,  expanded  DLR  carcass  tracking  functionality  provides  improved 
communications  up  and  down  the  chain  of  command  in  order  to  limit  unnecessary  costs. 

The  Fiscal  Year  2009  requirement  was  projected  at  $996,329.  The  statement  of 
work  also  calls  for  the  following  five  personnel  positions  to  be  provided  by  the 
contractor: 

•  One  Program  Manager 

•  One  Software  Application  Engineer  II 

•  One  Software  Application  Engineer  III 

•  One  Project  Control  Technician 

•  One  Web  Technician  II  (Fleet  and  Industrial  Supply  Center — 
Norfolk,  2008) 

The  contractor  satisfies  this  by  providing  five  full-time  programmers,  with  one 
out  of  the  five  dedicated  to  submarine  force  CMP  development. 
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5.  Summary 

CMP,  a  $1  million  per  year  program,  improves  Fleet  readiness,  while  reducing 
man-hours  spent  on  tedious  data  entry  and  analysis.  It  does  this  by  transmitting  key 
logistics  data  from  the  Fleet  to  shore  leadership  to  ensure  that  the  staff  objectives  are 
continually  met. 

C.  CNSF  WEB 

CNSF  Web  is  a  knowledge  management  system.  It  “provides  CNSF  and  its  ships 
and  subordinate  commands  (afloat  and  ashore)  with  a  secure,  centralized,  content-driven, 
integrated  web-based  collaboration  system”  (Commander,  Naval  Surface  Forces,  2008e). 

1.  System  Background 

The  original  development  for  CNSF  Web  stems  from  a  U.S.  Pacific  Fleet  staff 
knowledge  management  study  conducted  in  1999.  The  study  found  through  requirements 
analysis  processes  that  a  government  owned,  Web-based,  collaboration  toolset  to  provide 
a  central  location  for  all  organizational  knowledge  was  desirable  for  mission  support. 
Specific  requirements  included  the  ability  to  be  searchable  across  domains  and  be  both 
Nonsecure  Internet  Protocol  Router  Network  (NIPRNet)  and  Secure  Internet  Protocol 
Router  Network  (SIPRNet)  capable.  In  late  2000,  funded  by  its  staff  budget,  the  CNSP 
Knowledge  Manager  established  the  CNSF  Web,  built  upon  the  Microsoft  SharePoint 
Portal  System. 

In  2004,  the  first  consolidated  CNSP  and  CNSL  Web  portal  was  launched.  In 
tandem  with  the  Navy  Surface  Warfare  Enterprise  initiative,  subordinate  commands  were 
provided  the  option  of  consolidation  with  the  CNSF  Web  portal. 

The  changes  required  to  bring  a  single  site  together  for  both  coasts  was  not 
without  its  problems.  While  combining  the  information  technology  support  staffs  was 
deemed  effective,  there  remained  a  lack  of  automation  within  the  site.  This  led  to  a 
period  of  disuse  and  key  functional  areas  ignored  the  new  portal. 

In  May  2005,  Naval  Network  Warfare  Command  (NETWARCOM)  promulgated 

operation  CYBER  Condition  Zebra.  Its  goal  was  to  improve  the  overall  security  of  Navy 
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computer  networks  (DONCIOmemoFY06expendtiures,  05).  In  response,  CNSF  made 
two  major  shifts  to  the  CNSF  Web  system.  First,  it  absorbed  into  its  environment  several 
legacy  CNSF  information  technology  systems,  such  as  CMP.  Second,  in  2006,  it 
transferred  to  the  NMCI  Enterprise  Network.  At  that  time,  a  plan  was  detailed  to  shift 
CNSF  Web  to  the  CUSFFC  Fleet  Forces  Online  (FFO)  Portal. 

In  2007,  CNSF  mandated  the  migration  of  all  subordinate  public  and  private 
websites  to  its  purview.  This  was  in  response  to  NAV ADMIN  145/07,  which  stated  its 
objectives  to  “move  the  navy  toward  a  single  investment  strategy  for  websites,  provide 
enhanced  website  information  assurance  and  accountability,  and  establish  Navy  website 
standardization  and  branding”  (NAV ADMIN,  2007). 

NETWARCOM  then  accredited  CNSF  Web  NIPRNet  with  the  Authority  to 
Operate  (ATO)  in  July  2008.  Currently,  CNSF  Web  SIPRNet  is  awaiting  Interim 
Authority  to  Operate  from  its  certifying  authority. 

2.  Mission  Need 

The  primary  mission  support  function  of  the  CNSF  Web  is  force  alignment.  It 
does  so  by  linking  the  primary  staff  functions  of  all  echelons  within  the  chain  of 
command,  from  the  deckplate  level  to  the  Commander,  Naval  Surface  Forces,  in  one 
electronic  location.  It  does  so  by  providing  the  resource  for  staffs  to  publish  policies  and 
directives,  follow  various  operational  metrics,  and  communicate  with  each  other  on 
pertinent  topics. 

In  addition,  it  is  the  primary  tool  utilized  by  CNSF  to  meet  the  CNO’s  FORCEnet, 
“Alignment”  and  Sea  Enterprise  goals  along  with  the  business  requirements  of  the 
Surface  Forces  Enterprise  (SWE). 

Finally,  it  provides  a  true,  Enterprise  solution  for  the  claimancy.  As  a  certified 
Enterprise  Level  Collaboration  Portal,  it  supports  the  Navy  Enterprise  and  SWE  efforts  to 
reduce  inefficiencies  and  redundancies  and  leverage  Enterprise  systems  to  create  cost 
savings  for  the  SWE  (PPT  Brief  Contract  Consolidation). 
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3. 


Technical  Requirements 


Users  access  the  CNSF  Web,  diagrammed  in  Figure  15,  via  any  Web  browser. 
The  uniform  resource  locator  (URL)  for  the  publically  accessible  unclassified  site  is 
http://www.surfaceforces.surfor.navy.mil/default.aspx.  The  URL  for  the  NIPRNet  site  is 
https://www.surfor.navy.mil/,  and  the  classified  SIPRNet  URL  is 
https://www.surfor.navy.smil.mil/. 
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Figure  15.  CNSF  Web  Architecture 

Users  accessing  publically  accessible  Web  sites,  such  as  family  support  pages  or 
Public  Affairs  Officer  (PAO)  sites,  are  directed  to  the  Web  servers  at  the  NMCI  DMZ 
facility  building  W143  located  onboard  Norfolk  Naval  Station. 

Users  wishing  to  access  TORIS,  CMP,  or  other  unclassified  data,  need  to  clear 
SSL  and  PKI  controls  first.  Then,  they  may  access  the  web  and  data  servers  on  the 
unclassified  network.  All  operationally  classified  information  is  directed  to  the  Web  and 
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data  servers  on  the  SIPRNet.  Both  of  these  groups  of  servers  are  located  in  Building  NH- 
13,  also  on  Norfolk  Naval  Station  (PPT  Brief:  SURFOR  Web  Overview). 

4.  System  Description 

As  stated  earlier,  the  CNSF  Web  provides  the  claimancy  with  a  “secure, 
centralized,  content-driven,  integrated  web-based  collaboration  system”  (PPT  Brief: 
SURFOR  Web  Overview).  In  addition  to  each  CNSF  staff  department  Web  site,  it 
houses  all  afloat  command  Web  sites,  extranets,  CLASSRON,  Assault  Craft  Unit 
homepages,  and  the  Web  sites  for  all  SWE  cross  functional  teams.  All  in  all,  there  are 
over  4,000  site  administrators  distributed  throughout  the  system. 

The  Web  is  a  resource  for  the  conduct  of  key  CNSF  business  processes.  Those 
include: 

•  Publicly  Accessible  Web  sites 

•  Family  Web  sites 

•  Private  Business  Collaboration  sites 

•  Departure  From  Specifications  Database  (DFS) 

•  C5  Readiness  Assessment  (C5RA) 

•  Authorized  Equipment  Listing  Program  (AEL) 

•  COMET  II 

•  Hot  Wash 

•  2MCAL 

•  War  Fighting  Improvement  Program  (WFIP) 

•  Continuous  Monitoring  Program  (CMP) 

•  ATGLANT  Toolkit  and  Level  of  Knowledge  Program  (LOK) 

•  (PPT  Brief:  SURFOR  Web  Overview) 

Responsibilities  for  the  system  management  are  divided  as  follows:  CNSL  N6 
performs  system  engineering,  network  administration,  database  administration,  and  web 
development  functions.  CNSP  N6  focuses  on  program  management,  Web  development 
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strategy,  and  SharePoint  specific  functions,  such  as,  technical  architecture  development, 
user  support,  and  business  analysis  and  training. 


Specific  contract  costs  for  FY09  are  broken  out  as  follows: 


Dollar  Value 

Location 

Purpose 

Contract  Manager 

$703,000 

San  Diego,  CA 

CNSF  Web 

Support 

US  Army  CECOM,  Fort 

Monmouth 

$703,580 

Norfolk,  VA 

CNSF  Web 

Support 

US  Army  CECOM,  Fort 

Monmouth 

$672,120 

Norfolk,  VA 

NMCI  Support 

US  Army  CECOM,  Norfolk 

$156,863 

San  Diego,  CA 

NMCI  Support 

NAVICP,  Mechanicsburg 

$2,235,563 

TOTAL 

Table  2.  CNSF  Web  FY09  Funding  Structure 


This  funding  structure  breaks  out  four  distinct  groups  within  the  purview  of  the 
system.  First  and  second  are  the  actual  Web  server  maintenance  contracts  for 
information  technology  support  on  the  West  Coast  and  East  Coast.  Specific  duties  and 
responsibilities  for  those  contracts  include:  hardware  configuration  and  maintenance; 
software  installation,  modification/correction,  and  security;  database  administration  and 
management;  training  in  a  one-on-one,  webinar,  and/or  classroom  application 
environment,  as  well  as  workstation  end-user  training.  See  Figure  16. 
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SURFORWEB  Support  Roles  and  Cost 


CNSP  Personnel 

Program  Manager  (Civilian)  1  man  yr 

-  Provides  Knowledge  Management  and  Information  Management 
support  for  SURFOR  N6.  Manages  SURFOR  Web  Program 
and  provides  technical  support  for  SURFOR  Web  contract. 
Member  of  SWE  Overarching  Metrics  Team  (OMT)  and  provides 
technical  support  for  SWE  Data  Warehouse  contract.  Leads 
NIAPS  Replication  BRT  for  SWE. 

SURFOR  Web  Content  Manager  (Civilian)  1  man  yr 

Provides  Content  Management  support  for  SURFOR  Web. 
Provides  database  analysis  expertise.  Provides  Q/A  and  testing 
functions.  Serves  as  backup  for  Program  Manager. 

SharePoint  Technical  Architect  (Contractor)  1  man  yr 

-  Provides  overarching  technical  solutions  for  SURFOR  Web  and 
leads  contractor  supports  team.  Serves  as  backup  to  Content 
Manager. 

SharePoint  Analyst  (Contractor)  1  man  yr 

Provides  SharePoint  solutions  for  SURFOR  Web.  In  charge  of 
database  external  inputs  into  SURFOR  Web  including  COMET. 
Serves  as  backup  to  Technical  Architect  and  to  Content 
Manager. 

Training  Coordinator  (Contractor)  1  man  yr 

-  Provides  SharePoint  Training  and  Business  Process  Analysis 
support.  Training  is  both  online,  one  and  one  and  classroom. 
Serves  as  backup  to  User  Support. 

SharePoint  User  Support  (Contractor)  1  man  yr 

Provides  user  support  and  site  administration  for  SURFOR  Web. 
Provides  web  graphic  design  and  deployment  support.  Serves 
as  backup  to  Training  Coordinator. 

COMBINED  CNSP  SUPPORT  LEVEL  OF  EFFORT  =  6  Man  Years 


Years  of  Core  CNSP  Business  Knowledge:  80+  years 

CNSP  FUNDED  COSTS 
CTR  Cost:  $760K  per  annum 
SW/HDW  Cost:  $125K  per  annum  (NIPR  &  SIPR) 


CNSL  Personnel 

Chief  Information  Officer  (Civilian)  0.15  man  yr 

Provides  oversight  on  contracts,  direction  and  supervision  to  the 
SURFORWEB  development  and  engineering  team  providing  IT 
support  services 

Information  Assurance  Manager  (Civilian)  0.25  man  yr 

-  Ensures  that  the  Information  Assurance  requirements  of 
SURFORWEB  are  addressed  and  is  operated  in  accordance  with 
DOD/DON  directives 

Configuration  Manager/Certification  Agent  (CTR)  1  man  yr 

-  Manages  the  SURFORWEB  Configuration  Management  functions 
and  documentation  to  ensure  system  baselines  are  maintained. 
Conducts  C&A  reviews  for  system  accreditation.  Direct  advisor  to 
CNSL  N64  on  all  technical  and  program  requirements 

System  Network  Engineer/IAO  (CTR)  1  man  yr 

Manages  the  system/network  to  ensure  configuration  is  in 
accordance  with  DOD/DON  directives 
System  Network  Administrator  (CTR)  1  manyr 

Performs  system  and  network  administrative  functions  such  as 
system  configuration,  system  backup/restores,  and  system 
troubleshooting 

SharePoint  DBA  (CTR)  1  manyr 

Performs  web  development,  SQL  database  administration  and 
tuning,  automated  report  management  and  user  account 
m  an  ag  em  ent  /ad  m  i  n  i  strati  o  n 
Web  Master/Developer  (CTR)  1  man  yr 

-  Performs  web  development,  content  management  and  monitoring, 
user  administration  and  webmaster  training 

COMBINED  CNSL  SUPPORT  LEVEL  OF  EFFORT  =  5.4  Man  Years 

Years  of  Core  CNSL  Business  Knowledge:  80+  years 

CNSL  FUNDED  COSTS 

CTR  Cost:  $820K  per  annum  +  $10K  per  annum  training 
Infrastructure  Cost:  $75K  per  annum  (NIPR  &  SIPR  refresh) 


Figure  16.  CNSF  Web  Support  Roles  by  Coast 

The  third  and  fourth  line  items  provide  technical  support  for  the  NMCI  network 
within  the  CNSF  claimancy.  This  support  includes  network  engineering,  repair  services, 
system  analysis  and  technical  services,  computer  and  LAN  technical  support,  ordering, 
billing,  account  management  and  inventory  control  of  hardware  and  software,  and 
information  assurance  process  control  and  management,  and  video  teleconferencing 
support  (FY10  Budget  Document  Consolidated  IT  Support,  2009). 

Recent  data  shows  that  the  CNSF  Web  receives  373  average  requests  per  day. 
From  July  14  to  August  13,  2009,  1,517  distinct  users  accessed  the  system.  See  Figure 
17. 
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Figure  17.  CNSF  Web  Usage  Data  as  of  August  13,  2009 

5.  Summary 

CNSF  Web,  a  $2.2  million  per  year  program  aligns  the  surface  force  to  match  the 
objectives  set  by  CNSF.  It  accomplishes  this  by  providing  the  means  for  all  echelons  of 
command  within  the  organization  to  effectively,  efficiently,  and  securely  communicate 
via  the  Web. 

D.  SUMMARY 

This  chapter  provided  an  objective  description  of  the  three  major  information 
systems  that  this  project  evaluates.  Its  data  is  current  as  of  the  end  of  Fiscal  Year  2009. 
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IV.  COMMANDER,  NAVAL  SURFACE  FORCES  (CNSF) 
INFORMATION  SYSTEMS  RESEARCH 

This  chapter  evaluates  the  information  technology  systems  on  a  case  by  case 
basis.  It  does  so  first  by  delineating  the  quality  attributes  that  collectively  compose  a 
successful  information  system.  Later,  it  describes  the  evaluation  methodology  that  we 
used  to  review  the  systems.  Finally,  it  provides  a  comprehensive  evaluation  for  each 
information  technology  system,  based  on  user  feedback  and  Program  Manager  responses 
to  our  interview  questions. 

A.  ATTRIBUTES  OF  A  SUCCESSFUL  INFORMATION  SYSTEM 

Unlike  a  simple  math  problem,  there  is  no  single  right  answer  when  it  comes  to  an 
organization’s  search  for  an  information  system  solution.  There  are  a  multitude  of 
variables,  both  logical  and  physical,  specialized  in-house  or  generalized  commercial  off 
the  shelf,  technologically  cutting  edge  or  passed  down  as  a  legacy  product,  that 
management  can  choose  between  in  its  search  for  the  best  tool  for  the  job. 

However,  there  are  a  certain  set  of  characteristics  that  most  Department  of 
Defense  (DoD)  information  systems  share  in  common.  These  “quality  attributes,”  when 
planned  for  in  the  development  phase  of  the  system,  can  greatly  enhance  the  program’s 
ability  to  provide  the  desired  solution  correctly  and  efficiently  over  time.  Simply  put, 
systems  exhibiting  superior  delivery  of  these  traits  are  much  more  likely  to  make  the 
customer  happy. 

It  is  important  to  note  that  these  quality  attributes  can  be  described  on  a  scale  or 
range,  as  they  can  be  expressed  as  being  present  to  varying  degrees.  Like  functional 
attributes,  it  is  costly  to  improve  a  particular  quality  attribute.  Therefore,  organizations 
must  build  toward  the  desired  degree  of  each  quality  attribute  that  is  sufficient  for  its 
purposes. 

The  following  is  a  breakdown  of  the  six  key,  quality  attributes  that  can  be  easily 
distinguished  within  the  CNSF  information  systems. 
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1.  Usability 

Usability  is  the  relative  simplicity  that  end  users  experience  in  terms  of  every-day 
operation  of  the  system.  It  is  represented  by  an  easy  to  understand,  visually  appealing, 
and  intuitive  interface.  In  addition,  it  should  be  clear  to  the  users  that  the  functions  they 
are  performing  easily  relate  to  the  overall  goal  of  the  system  itself. 

2.  Interoperability 

Interoperability  is  a  necessary  quality  attribute  for  the  information  systems 
handled  by  most  large  organizations  such  as  CNSF.  It  describes  the  systems’  ability  to 
work  together  with  other  information  technology  systems.  This  is  important  not  only 
because  it  enables  the  automation  of  data  processing  across  platforms,  but  also  because  it 
makes  it  easier  for  decision  makers  to  find  and  utilize  the  information  that  they  need. 

Generally,  interoperability  is  described  as  a  technical  term,  within  a  systems 
engineering  context.  In  addition,  it  can  be  also  used  to  describe  the  information  system 
within  the  broader  context  of  the  organization’s  culture  as  a  whole.  For  example,  an 
organization’s  uses  of  paper  requests  for  personnel  travel-request  processing  and  an 
automated,  electronic  financial  accounting  information  technology  system.  An 
information  system  that  allows  for  Web-based  travel  processing  would  not  be  considered 
fully  interoperable  simply  because  it  feeds  required  data  into  the  financial  accounting 
system.  Instead,  management  must  ensure  that  personnel  are  using  the  system  quickly 
and  effectively.  A  culture  that  inhibits  change  to  the  new  system  restricts  its 
interoperability. 

3.  Responsiveness 

Responsiveness  is  a  straightforward  characteristic,  but  nonetheless  necessary  to 
the  success  of  any  information  technology  system.  It  relates  to  the  speed  with  which  it 
responds  to  user  interactions.  While  a  small  percentage  of  systems  possess  a  requirement 
for  real-time  data  flow,  the  time  criticality  of  many  systems  is  not  normally 
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the  highest  priority  for  management.  However,  failure  to  develop  responsiveness  within 
an  information  system  can  cause  several  unintended  consequences,  including  the  loss  of 
workforce  productivity. 

4.  Maintainability 

Maintainability  describes  an  information  system’s  capacity  for  improvement  and 
modification.  Change  to  the  system  can  be  requested  for  any  number  of  reasons,  but 
most  modifications  are  undergone  to  repair  faults,  satisfy  successive  management 
requirements,  support  subsequent  maintenance  efforts,  or  transform  to  interact  within  the 
latest  and  current  environment. 

5.  Scalability 

Scalability  relates  to  the  system’s  ability  to  support  an  increased  workload,  more 
users,  and/or  more  data,  in  a  timely  fashion.  The  more  robust  the  system,  the  more 
advantageous  scalability  is  to  management  because  it  is  more  likely  to  be  able  to 
successfully  provide  what  is  required  from  it.  In  addition,  an  information  system  exhibits 
even  better  scalability  when  it  provides  for  a  greater  workload  without  impacting  current 
functions. 


6.  Standardization 

Standardization,  the  final  quality  attribute  detailed  in  this  paper,  communicates 
the  matching  of  an  information  system’s  technical  standards  and  specifications  to  those 
established  by  higher,  legitimate,  or  de  facto  authority.  In  this  context,  the 
standardization  of  the  system  should  match  those  requirements  set  by  the  DoD  and  the 
Department  of  the  Navy  (DoN)  Chief  Information  Officers  (CIOs).  One  example  of  a 
standard  is  found  in  DoD  Instruction  8510.01,  dated  November  28,  2007.  It  dictates  the 
necessity  for  DoD  Components  to  “Operate  only  accredited  information  systems  (i.e., 
those  with  current  Authority  to  Operate  (ATO),  interim  authorization  to  operate,  or 
interim  authorization  to  test)”  (DoD  8510.01).  This  mandate  ensures  that  only 
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information  technology  systems  that  pass  an  approved  certification  process,  which 
reviews  broad  system  architecture  and  information  assurance  capabilities,  operate  within 
DoD. 


B.  EVALUATION  METHODOLOGY 

The  following  section  describes  the  method  by  which  we  conducted  our 
evaluation.  It  details  the  primary  reference  for  our  review  as  well  as  the  key  performance 
indicators  that  we  based  our  review  upon.  Finally,  it  explains  the  actual  method  that  we 
used  to  retrieve  the  information  necessary  for  this  project. 

1.  Review  of  the  Primary  Evaluation  Reference 

Our  primary  reference  for  conducting  the  baseline  assessment  for  information 
systems  at  CNSF  was  the  Department  of  the  Navy  Information  Technology  Investment 
Evaluation  Handbook.  Originally  conceived  to  offer  components  with  a  solid  framework 
of  questionnaires  and  surveys  for  evaluating  their  prospective  information  technology 
investments,  it  is  easily  adapted  to  our  purpose  of  conducting  a  baseline  assessment  of 
installed  systems.  In  fact,  it  recommends  that  managers  continue  to  evaluate,  at  least 
once  every  three  years,  that  their  programs  meet  the  current  needs  of  the  organization  by 
adjusting  the  provided  steps  to  better  apply  to  the  applicable  review. 

The  reference  describes  that  a  completed  review  should  result  in  the  manager 
determining  whether  the  program  was  a  success  or  failure.  The  data  the  evaluation 
provides  describes  the  program’s  progress  of  meeting  its  cost,  schedule,  and  performance 
goals  by  focusing  on  five  core  concepts:  Mission,  Performance,  Management,  Financial, 
and  Technical.  The  following  section  describes  each  key  area  in  detail. 

a.  Mission 

The  mission  area  details  the  link  between  the  system  itself  and  the 
organization’s  overarching  goals  and  objectives.  In  a  resource  constrained  environment 
such  as  the  one  that  CNSF  is  coping  with  today,  the  link  between  investment  and 
strategic  mission  must  be  clear  and  forthright.  Otherwise,  the  return  on  investment  of  an 

alternative  project  based  on  the  needs  of  the  unit  would  most  likely  be  greater. 
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The  evaluator  should  be  sure  to  draw  distinct  correlations,  or  in  the  case  of 
a  failure,  the  lack  thereof,  between  the  information  technology  system’s  intended 
objectives  and  the  overall  needs  of  the  organization,  in  our  case,  mission  readiness  and 
mission  performance. 

The  recommended,  primary  method  for  the  evaluation  of  the  mission 
concept  is  through  the  interview  of  end  users.  This  allows  the  reviewer  to  gather 
firsthand  information  on  how  well  the  system  supports  their  specified  duties  and 
responsibilities,  which  are  aligned  with  those  of  higher  authority. 

b.  Performance 

The  performance  concept  is  based  on  the  review  of  predefined  technical 
and  functional  Key  Performance  Indicators  (KPIs).  During  development,  these  data 
points  represent  the  intended,  objective  benefits  of  the  system.  Upon  deployment  of  the 
system,  the  KPIs  are  reviewed  to  ensure  that  the  system  is  matching  or  exceeding  the 
original  intended  business  and  performance  requirements  established  by  management. 

The  goal  of  the  performance  review  is  twofold.  First,  it  explains  whether 
or  not  the  system  is  actually  effective  in  its  original  intended  purpose.  Second,  it  forms 
the  basis  for  a  review  of  the  program’s  management  team.  A  failure  to  meet  KPIs  is  a 
failure  on  management’s  ability  to  direct  the  system  over  time. 

KPIs  fall  under  one  of  two  different  categories — functional  and  technical. 
The  former  evaluates  the  system’s  worth  to  the  organization  and  its  significance  to  the 
end  users  based  on  how  well  it  does  its  job.  Ideally,  this  category  is  composed  of  both 
quantitative  and  qualitative  parameters.  Technical  KPIs  focus  on  quantitative 
measurements  of  the  physical  capabilities  of  the  system  itself.  Examples  of  this  include 
latency  times  and  the  maximum  number  of  user  accounts. 

Technical  KPIs  are  important,  but  in  and  of  themselves,  do  not  provide 
adequate  data  for  the  evaluation.  Instead,  it  is  up  to  management  to  determine  the  value 
to  the  organization’s  mission  that  is  represented  by  the  technical  KPIs. 
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c.  Management 

Management  is  the  term  that  describes  the  review  of  the  effectiveness  of 
the  system’s  Program  Manager.  Specifically,  it  details  the  development  and  enforcement 
of  related  directives,  policies,  training,  and  other  aspects  of  the  program  administration. 

A  more  in-depth  evaluation  of  the  management  area  focuses  on  how  well 
the  system  meets  applicable  Department  of  Defense  and  Department  of  the  Navy  written 
standards  and  requirements. 

d.  Financial 

The  reference  describes  the  financial  concept  as  the  section  that  describes 
the  balance  between  the  system’s  total  cost,  its  benefits,  and  its  return  on  investment. 

Total  cost,  while  very  important,  is  not  the  only  data  point  included  in  this 
analysis.  This  is  because  total  cost  does  not  provide  a  complete  comparison  between 
programs.  Instead,  the  differences  between  the  actual  benefits,  both  tangible  and 
intangible,  realized  by  the  organization  should  be  compared  as  well.  For  example,  a 
prospective  single,  information  system  with  a  $6  million  total  ownership  cost  may 
promise  to  replace  two  legacy  systems,  which  combine  to  cost  $5  million  and  fail  to 
communicate  with  each  other.  Considering  cost  alone,  the  replacement  proposal  would 
fail.  However,  if  one  considered  the  productivity  increase  that  would  present  itself  by 
replacing  the  legacy  systems  with  one  cohesive  system,  management  would  be  more 
likely  to  approve  the  change.  Therefore,  it  is  essential  that  the  review  includes  the 
relation  between  total  cost  and  the  system’s  mission  contributions. 

e.  Technical 

This  area  determines  the  technical  effectiveness  of  the  system.  It  does  so 
by  analyzing  user  feedback  and  test  results  of  actual  system  performance.  The  goal  is  to 
conclude  whether  the  operational  readiness  of  the  system  is  satisfactory  or  not. 
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2. 


Actual  Interview 


Specifically,  for  this  research,  we  conducted  two  types  of  interviews.  First,  we 
met  with  representatives  from  each  specific  system’s  management  to  gain  an 
understanding  on  the  objectives,  workings,  and  overhead  related  to  each  program. 
Second,  we  interviewed  end  users,  above  the  shipboard  level.  This  set  of  interviews  was 
designed  to  understand  what  day-to-day  impacts  each  system  had  on  its  customer. 

C.  FINDINGS  AND  FEEDBACK 

This  section  provides  three  pieces  of  information  on  each  information  technology 
system.  First,  it  discusses  the  information  we  received  from  the  program  manager’s  of 
each  system  in  terms  of  the  five  core  concepts  contained  within  the  Department  of  the 
Navy’s  Information  Technology  Investment  Evaluation  Handbook.  Second,  it  describes 
the  user  feedback  that  we  received. 

1.  Training  and  Operational  Readiness  Information  Services  (TORIS) 
a.  Program  Manager  Research 

Despite  TORIS ’s  emphasis  on  training  vice  an  operational  focus,  it  still 
meets  the  objectives  for  mission  improvement  because  it  satisfactorily  supports  the 
continuous  training  requirements  established  by  SHIPTRAIN.  This  reduces  the 
shipboard  readiness  “bathtub  effect”  experienced  when  ramp  down  of  readiness  coming 
off  of  deployment  is  followed  by  the  re-training  and  ramp  up  leading  into  a  subsequent 
deployment.  In  addition,  that  improves  CNSF’s  ability  to  meet  its  deployment 
requirements  as  designated  by  Commander,  U.S.  Fleet  Forces  Command’s  (CUSFFC) 
6+2  plan.  However,  the  only  real  productivity  improvements  are  administrative  in 
nature,  as  the  system  automates  much  of  the  tedious  data  collection,  tracking,  and 
reporting  requirements  up  the  chain  of  command.  Despite  this  fact,  due  to  the  maturity  of 
the  program,  it  sufficiently  supports  the  Surface  Force  Training  Manual  (SFTM) 
requirements. 

Unfortunately,  there  are  no  contractual  key  performance  indicators  for  this 
program,  other  than  the  few  requirements  built  into  the  contract  statement  of  work  and 
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listed  in  Chapter  III,  used  to  measure  either  the  system  or  the  contractor’s  performance. 
This  weakness  will  be  addressed  in  Chapter  V. 


Since  the  casualty  of  the  system  will  not  preclude  a  ship’s  ability  to  get 
underway,  it  is  considered  mission  enhancing  for  afloat  units.  However,  it  is  mission 
critical  for  afloat  units’  training  programs,  as  the  use  of  TORIS  is  required  by  the  SFTM. 
In  addition,  it  is  critical  to  the  mission  of  the  Afloat  Training  Groups  (ATGs),  who  use  it 
extensively  in  order  to  train  and  certify  the  Fleet  as  ready  to  deploy. 

TORIS  is  not  an  official  program  of  record,  and  changes  to  the  system  are 
handled  by  a  strict  process.  First,  proposed  changes  are  vetted  through  a  Configuration 
Control  Group  that  is  composed  of  the  ATG  Commanding  Officers,  the  CNSF  Assistant 
N7,  and  the  CNSL  Assistant  N7.  Then,  all  changes  recommended  for  approval  are  routed 
to  a  Configuration  Control  Board  for  a  final  decision.  The  Squadron  Commodores, 
CNSF  N7,  and  the  CNSL  N7  collectively  sit  that  board. 

There  are  no  financial  goals  established  for  this  program,  which  is  a 
weakness  that  could  be  possibly  addressed  in  terms  of  return  on  investment. 

Finally,  the  system’s  information  technology  infrastructure  does  not 
present  any  significant  performance  issues  that  require  immediate  attention.  However, 
over  time,  as  the  size  of  the  databases  expand,  so  does  its  requirements  for  storage. 
Developers  estimate  that  within  two  years,  they  will  be  forced  to  delete  the  oldest 
archived  data  to  prevent  the  system  from  falling  into  partial  mission  capability.  This  is 
because  the  current  database  servers  each  possess  only  500GB  of  hard  drive  space.  A 
possible  improvement  would  be  to  shift  to  a  Storage  Area  Network  (SAN)  solution. 

No  formal  end-user  training  program  is  in  place  for  TORIS,  although  the 
training  of  new  ATG  staff  personnel  is  administered  as  required.  A  unified  training 
program  between  the  coasts,  conducted  or  managed  by  the  User  Training  Services 
requirement  within  the  contract’s  statement  of  work,  would  resolve  this  concern. 

Finally,  system  developers  report  zero  remaining  redundancies  within  the 
system  based  on  the  merger  between  the  East  Coast  and  West  Coast  staffs. 
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b. 


User  Feedback 


The  following  tables  capture  the  User  Survey  results  for  TORIS.  The 
quantitative  number  of  responses  received  was  adequate  based  on  the  numbers  of  surveys 
that  were  distributed  by  ATG.  The  survey  responses  received  were  from  senior  enlisted 
Active  Duty  military  members. 
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1 .  Ease  of  access  to  system  into  the 

collection  devices. _ 

2.  Ease  of  access  into  the  ships  system  or 

to  pass  to  the  TLO. _ 

3.  Is  the  TORIS  system  installed  on  the 

laptops  easy  to  use? _ 

4.  Is  the  MSP  system  installed  on  the 
Extranet  easy  to  use. 


5.  Have  confidence  that  the  system  is 

working  correctly. _ 

6.  Degree  to  make  input  for  changes  to  the 

system. _ 

7.  Collection  device  is  current. 

8.  Software  installed  on  your  computer  is 

the  most  recent. _ 

9.  Does  your  system  operate  well  with  the 

ships  IT21  system? _ 

10.  Is  the  time  to  enter  data  cards  into  your 

system  minimum? _ 

1 1 .  Are  there  contingency  procedures  when 

your  computer  is  down? _ 

12.  Are  there  contingency  procedures  when 
the  TLO’s  system  is  not  working? 


13.  Is  the  system  responsive  to  changing 

user  needs? _ 

14.  Does  the  system  produce  professional 

reports? _ 

15.  Are  the  report  data  accurate? _ 

16.  Fast  response  time  from  helpdesk  to 
remedy  issues? 

17.  Positive  attitude  from  Help  Desk  to 

users. _ 

18.  Fast  response  time  from  Help  Desk 

staff  to  remedy  problems. _ 

19.  System’s  ability  to  improve  users’ 

personal  productivity. _ 

20.  Extent  of  user  training. 


21.  System’s  ability  to  enhance  the  learning 
experience  of  the  user. 


22.  Documentation  to  support  user  training 
is  provided. 


23.  System  enhances  warfighter  readiness. 


24.  System  provides  tangible  financial 
benefit. 


Very 

Important 

Important 

Somewhat 

Important 

Not 

Important 

N/A 

12 

11 

1 

11 

13 

11 

11 

2 

14 

8 

2 

14 

7 

3 

12 

8 

1 

3 

10 

11 

3 

16 

8 

7 

5 

1 

11 

12 

9 

3 

12 

11 

1 

9 

9 

2 

2 

2 

10 

12 

2 

9 

14 

1 

13 

11 

13 

7 

4 

12 

6 

3 

3 

12 

6 

3 

3 

12 

8 

3 
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14 

7 

3 

8 

12 

4 

10 

9 

2 

3 

12 

7 

4 

1 

7 

8 

1 

2 

6 

-a  "d 

5— I  *-l-H 

(D  c/2  c/3 

^  o3  o3 

m  m 


_3 _ L 

_4 _ 2_ 

_5 _ 

3  3 


8  1 


\ _ 

2 _ 

2 _ 3_ 

2 _ 3_ 

8  3 


2 _ 8_ 

7  3 

5  6 


4  2 

8  3 


Table  3.  TORIS  User  Feedback  Numerical  Responses 
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Dissatisfied 


1 .  Ease  of  access  to  system 
into  the  collection  devices. 

2.  Ease  of  access  into  the 

ships  system  or  to  pass  to 
the  TLO. _ 

3.  Is  the  TORIS  system 

installed  on  the  laptops 
easy  to  use? _ 

4.  Is  the  MSP  system  installed 
on  the  Extranet  easy  to  use. 


5.  Have  confidence  that  the 

system  is  working 
correctly. _ 

6.  Degree  to  make  input  for 

changes  to  the  system. _ 

7.  Collection  device  is  up  to 

date. _ 

8.  Software  installed  on  your 

computer  is  the  most 
recent. _ 

9.  Does  your  system  operate 

well  with  the  ships  IT21 
system? _ 

10.  Is  the  time  to  enter  data 

cards  into  your  system 
minimum? _ 

1 1 .  Are  there  contingency 

procedures  when  your 
computer  is  down? _ 

12.  Are  there  contingency 
procedures  when  the 
TLO’s  system  is  not 
working? 


Very 

Important 

Important 

Somewhat 

Important 

Not 

Important 

N/A 

Very 

Satisfied 

Satisfied 

Somewhat 

Satisfied 

Dissatisfied 

N/A 

50% 

46% 

4% 

0% 

0% 

4% 

79% 

13% 

4% 

0 

% 

46% 

54% 

0% 

0% 

0% 

21% 

54% 

17% 

8% 

0 

% 

46% 

46% 

8% 

0% 

0% 

13% 

58% 

21% 

0% 

8 

% 

58% 

33% 

8% 

0% 

0% 

8% 

58% 

13% 

13 

8 

% 

% 

58% 

29% 

13% 

0% 

0% 

13% 

50% 

33% 

4% 

0 

% 

50% 

33% 

4% 

0% 

13 

4% 

38% 

25% 

17 

17 

% 

% 

% 

42% 

46% 

0% 

0% 

13 

17% 

63% 

8% 

4% 

8 

% 

% 

67% 

33% 

0% 

0% 

0% 

46% 

50% 

4% 

0% 

0 

% 

29% 

21% 

0% 

4% 

46 

17% 

29% 

8% 

0% 

46 

% 

% 

50% 

38% 

13% 

0% 

0% 

4% 

54% 

29% 

13 

0 

% 

% 

50% 

46% 

0% 

0% 

4% 

4% 

46% 

29% 

13 

8 

% 

% 

38% 

38% 

8% 

8% 

8% 

4% 

38% 

33% 

13 

13 

% 

% 

productivity. 

20.  Extent  of  user  training. 

58% 

29% 

13% 

0% 

0% 

8% 

46% 

33% 

13 

0 

% 

% 

21.  System’s  ability  to  enhance 

33% 

50% 

17% 

0% 

0% 

4% 

50% 

42% 

4% 

0 

the  learning  experience  of 

% 

the  user. 

22.  Documentation  to  support 

42% 

38% 

8% 

13% 

0% 

0% 

42% 

38% 

13 

8 

user  training  is  provided. 

% 

% 

23.  System  enhances 

50% 

29% 

17% 

0% 

4% 

8% 

38% 

29% 

21 

4 

warfighter  readiness. 

% 

% 

24.  System  provides  tangible 

29% 

33% 

4% 

8% 

25 

4% 

46% 

25% 

8% 

17 

financial  benefit. 

% 

% 

Table  4.  TORIS  User  Feedback  Percentage  Responses 


Results  of  respondent’s  experience  and  usage: 


|  How  long  have  you  been  in  the  current  position?  (Number  of  respondents/percentage) 

1 

|  Less  than  1  year 

1-3  years 

3-5  years 

More  than  5  years 

8/33% 

15/63% 

1/4% 

0/0% 

Approximately  how  many  hours  a  day  do  you  use  this  system  in  the  performance  of  your  duties?  (Number 
of  respondents/percentage) 

Almost  never 

<  1  hour/day 

1-2  hours/day 

3-4  hours/day 

5-6  hours/day 

>  6  hours/day 

0/0% 

11/46% 

10/42% 

2/8% 

1/4% 

0/0% 

Table  5.  TORIS  User  Feedback  Respondent  Data 


Specific  user  feedback  was  generally  positive.  For  the  entire  list  of 
negative,  positive,  and  other  comments,  see  Appendix  A. 

2.  Continuous  Monitoring  Program  (CMP) 
a.  Program  Manager  Research 

The  stated  goals  in  the  program  justification  for  mission  improvement  are 
achieved  in  two  ways.  First,  CMP  clearly  provides  benefit  to  TYCOM  and  higher  DoN 
leadership  by  providing  views  of  the  operational  readiness  of  supply  departments  across 
the  fleet.  In  addition,  it  also  acts  as  a  valuable  tool  to  assess  and  correct  deficiencies 
within  afloat  supply  departments.  CMP  achieves  this  through  its  traffic  light  display  of 
pulse  points  deemed  important  by  TYCOM  staff.  This  display  is  powerful,  yet  concise 
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snapshot  of  the  given  unit’s  performance  and  areas  that  may  require  additional  attention. 
Additionally,  the  productivity  improvements  for  supply  personnel  is  significant  as 
representatives  from  S-l,  S-2,  S-3,  and  S-4  divisions  can  instantly  determine  the  posture 
of  the  division.  CMP  provides  this  analysis  much  faster  than  individually  pulling  manual 
reports  in  each  information  system.  Therefore,  the  system  is  currently  meeting  the 
mission  need  as  it  exists  now,  and  due  to  its  agility,  it  can  be  modified  to  quickly  add 
additional  pulse  points,  as  required. 

Like  TORIS,  there  are  no  contractual  key  performance  indicators  for 
CMP,  other  than  the  few  requirements  built  into  the  contract  statement  of  work  and  listed 
in  Chapter  III,  used  to  measure  either  the  system  or  the  contractor’s  performance.  This 
weakness  will  be  addressed  in  Chapter  V. 

Since  the  casualty  of  the  system  will  not  preclude  a  ship’s  ability  to  get 
underway,  CMP  is  considered  mission  enhancing  for  afloat  units,  even  though  its  use  is 
required  by  the  Surface  Force  Supply  Procedures  (SURFSUP).  In  addition,  it  is 
enhancing  to  the  mission  of  the  ATGs,  Class  Squadrons  (CLASSRONs),  Navy  Food 
Management  Team,  and  CNSF  staff,  that  use  it  for  pulse  point  monitoring  and  financial 
information. 

Changes  to  the  CMP  software  system  occur  often  and  frequently,  in 
response  to  primary  stakeholders  within  the  CNSF  and  CNSL  N41  staffs,  as  well  as  the 
NOOF  department.  Change  requests  are  submitted  to  the  individual  to  whom  CMP 
contractors  report  to:  CNSL  N411,  the  N411 — Supply  Logistics/Readiness  Officer. 

There  are  no  financial  goals  established  for  CMP,  which  is  a  weakness 
that  could  be  possibly  addressed  in  terms  of  return  on  investment. 

No  performance  issues  related  to  the  information  technology  infrastructure 
were  discovered  during  our  research  or  interviews,  as  the  program  currently  meets  all 
operational  requirements. 

Shore  establishments  train  various  stakeholders  on  CMP.  CNSP  and 
CNSL  N4  both  provide  an  overview  of  capabilities  to  prospective  Commanding  Officers 
and  prospective  Executive  Officers  enroute  to  ships.  The  Navy  Supply  Corps  School  also 
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provides  training  to  Supply  Officers  during  the  Basic  Qualification  Course  and  Supply 
Officer  Department  Head  Course.  Finally,  Chapter  17  of  the  CNSF  4440.1  provides  an 
extensive  list  of  instructions  for  end  users. 

Finally,  no  remaining  redundancies  exist  within  the  system  based  on  the 
merger  between  the  east  and  west  coast  staffs. 

b.  User  Feedback 

The  following  tables  capture  the  User  Survey  results  for  the  Continuous 
Monitoring  Program  (CMP).  The  quantitative  number  of  responses  received  was  far 
below  anticipated  expectations,  this  despite  several  follow-up  emails  and  phone  calls. 
The  survey  responses  received  were  from  Defense  contractors  from  two  CLASSRONs 
and  one  from  ATGPAC. 
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Ease  of  access  to  system. _ 

System  easy  to  use. _ 

Have  confidence  in  system. 
Degree  of  personal  control  over 
the  system. 


System  has  up  to  date  hardware. 
System  has  up  to  date  software. 
Interoperable  with  other 

systems. _ 

System’s  response  time. _ 

Contingency  procedures  when 
system  is  down. 

System  responsiveness  to 

changing  user  needs. _ 

Flexibility  of  the  system  to 
produce  professional  reports. 
Report  data  is  verifiable. 


Fast  response  time  from  support 
staff  to  remedy  technical  issues. 
Positive  attitude  from  Help 

Desk  to  users. _ 

Fast  response  time  from  Help 
Desk  staff  to  remedy  problems. 
Ability  of  the  system  to 
improve  users’  personal 

productivity. _ 

Extent  of  user  training. 

Ability  of  the  system  to 
enhance  the  learning  experience 

of  the  users’. _ 

Documentation  to  support  user 
training  is  provided. 


System  enhances  warfighter 

readiness. _ 

System  provides  tangible 
financial  benefit. 


Relevant  data  and  metrics  are 

captured  by  the  system. _ 

System  is  well  aligned  across 

the  CNSF  claimancy. _ 

System  supports  the  goals  of 
the  SWE. 


I  Importance  | 

Very 

Important 

Important 

Somewhat 

Important 

Not 

Important 

< 

£ 

2 

1 

3 

2 

1 

1 

2 

2 

1 

2 

1 

2 

1 

1 

2 

2 

1 

1 

1 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

1 

1 

2 

1 

1 

1 

2 

1 

2 

1 

2 

1 

1 

2 

Satisfaction 


Table  6.  CMP  User  Feedback  Numerical  Responses 
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Dissatisfied 


Im] 

portance 

Satisfaction 

ZD 

Very 

Important 

Important 

Somewhat 

Important 

Not 

Important 

N/A 

Very 

Satisfied 

Satisfied 

Somewhat 

Satisfied 

Dissatisfied 

N/A 

1 .  Ease  of  access  to 
system. 

67% 

33% 

0% 

0% 

0% 

67% 

33% 

0% 

0% 

0% 

2.  System  easy  to 
use. 

100% 

0% 

0% 

0% 

0% 

67% 

33% 

0% 

0% 

0% 

3.  Have  confidence 
in  system. 

67% 

33% 

0% 

0% 

0% 

67% 

33% 

0% 

0% 

0% 

4.  Degree  of 

personal  control 
over  the  system. 

33% 

67% 

0% 

0% 

0% 

33% 

67% 

0% 

0% 

0% 

5.  System  has  up  to 
date  hardware. 

67% 

0% 

0% 

0% 

33% 

33% 

33% 

0% 

0% 

33% 

6.  System  has  up  to 
date  software. 

67% 

0% 

33% 

0% 

0% 

33% 

33% 

33% 

0% 

0% 

7.  Interoperable  with 
other  systems. 

67% 

0% 

0% 

0% 

33% 

0% 

0% 

33% 

0% 

67% 

8.  System’s  response 
time. 

33% 

67% 

0% 

0% 

0% 

33% 

33% 

33% 

0% 

0% 

9.  Contingency 
procedures  when 
system  is  down. 

67% 

0% 

33% 

0% 

0% 

0% 

33% 

33% 

0% 

33% 

10.  System 

responsiveness  to 
changing  user 
needs. 

33% 

33% 

0% 

0% 

33% 

0% 

33% 

33% 

0% 

33% 

1 1 .  Flexibility  of  the 
system  to  produce 
professional 
reports. 

67% 

33% 

0% 

0% 

0% 

0% 

33% 

33% 

33% 

0% 

12.  Report  data  is 
verifiable. 

67% 

33% 

0% 

0% 

0% 

67% 

33% 

0% 

0% 

0% 

13.  Fast  response  time 
from  support  staff 
to  remedy 
technical  issues. 

67% 

0% 

0% 

0% 

33% 

67% 

0% 

0% 

0% 

33% 

14.  Positive  attitude 
from  Help  Desk  to 
users. 

67% 

0% 

0% 

0% 

33% 

67% 

0% 

0% 

0% 

33% 

15.  Fast  response  time 
from  Help  Desk 
staff  to  remedy 
problems. 

67% 

0% 

0% 

0% 

33% 

67% 

0% 

0% 

0% 

33% 

16.  Ability  of  the 
system  to  improve 
users’  personal 
productivity. 

67% 

33% 

0% 

0% 

0% 

33% 

33% 

0% 

33% 

0% 

17.  Extent  of  user 
training. 

67% 

0% 

0% 

0% 

33% 

33% 

33% 

0% 

0% 

33% 
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18.  Ability  of  the 
system  to  enhance 
the  learning 
experience  of  the 
users’. 

67% 

0% 

0% 

0% 

33% 

33% 

33% 

0% 

0% 

33% 

19.  Documentation  to 
support  user 
training  is 
provided. 

33% 

0% 

0% 

0% 

67% 

0% 

0% 

33% 

0% 

67% 

20.  System  enhances 
warfighter 
readiness. 

33% 

33% 

0% 

0% 

33% 

33% 

0% 

33% 

0% 

33% 

21.  System  provides 
tangible  financial 
benefit. 

67% 

0% 

33% 

0% 

0% 

67% 

0% 

33% 

0% 

0% 

22.  Relevant  data  and 
metrics  are 
captured  by  the 
system. 

67% 

33% 

0% 

0% 

0% 

33% 

33% 

33% 

0% 

0% 

23.  System  is  well 
aligned  across  the 
CNSF  claimancy. 

67% 

33% 

0% 

0% 

0% 

67% 

0% 

33% 

0% 

0% 

24.  System  supports 
the  goals  of  the 
SWE. 

33% 

67% 

0% 

0% 

0% 

33% 

33% 

33% 

0% 

0% 

Table  7.  CMP  User  Feedback  Percentage  Responses 


Results  of  respondent’s  experience  and  usage: 


|  How  long  have  you  been  in  the  current  position?  (Number  of  respondents/percentage) 

|  Less  than  1  year 

1-3  years 

3-5  years 

More  than  5  years 

1  /33% 

1  /  33% 

0/0% 

1/33% 

1 

Approximately  how  many  hours  a  day  do  you  use  this  system  in 
(Number  of  respondents/percentage) 

the  performance  of  your  duties? 

Almost  never 

<  1  hour/day 

1-2  hours/day 

3-4  hours/day 

5-6  hours/day 

>  6  hours/day 

0/0% 

0/0% 

3  /  100% 

_ 

0/0  % 

0/0% 

0/0% 

Table  8.  CMP  User  Feedback  Respondent  Data 


Overall,  user  feedback  was  positive,  but  limited.  For  the  entire  list  of 
negative,  positive,  and  other  comments,  see  Appendix  B. 
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3.  Commander,  Naval  Surface  Forces  (CNSF)  Web 

a.  Program  Manager  Research 

No  goals  for  mission  improvement  are  stated  in  the  program  justification. 
So,  the  program  has  few  documentable  references  to  how  it  improves  the  Command’s 
ability  to  meet  it  mission.  Like  a  bulletin  board,  it  is  used  for  informational  purposes 
only,  and  it  is  only  as  good  as  the  currency  and  relevance  of  the  information  posted,  as 
well  as  how  often  end  users  use  it  to  retrieve  that  information.  When  used  properly 
though,  it  is  a  keen  force  alignment  tool  good  for  spreading  policy  efficiently,  as  it 
reduces  the  need  for  redundant  messaging. 

Like  TORIS,  there  are  no  contractual  key  performance  indicators  for 
CNSF  Web,  other  than  the  few  requirements  built  into  the  contract  statement  of  work  and 
listed  in  Chapter  III,  used  to  measure  either  the  system  or  the  contractor’s  performance. 
This  weakness  will  be  addressed  in  Chapter  V. 

Since  the  casualty  of  the  system  will  not  preclude  a  ship’s  ability  to  get 
underway,  CNSF  Web  is  considered  mission  enhancing  for  afloat  units,  and  there  is  no 
written  policy  requiring  its  use.  In  addition,  it  is  enhancing  to  the  mission  of  the  shore 
commands,  who  use  it  solely  for  information  dissemination  purposes. 

Innumerable  changes  occur  to  the  CNSF  Web  regularly,  as  designated  site 
administrators  retain  both  the  privilege  and  responsibility  to  update  their  respective  areas 
of  concern.  Major  change  requests,  such  as  the  request  for  a  new  domain,  are  sent  to  the 
CNSF  Web  development  team. 

No  financial  goals  are  established  for  the  CNSF  Web,  which  is  a  weakness 
that  could  be  possibly  addressed  in  terms  of  return  on  investment. 

No  performance  issues  related  to  the  information  technology  infrastructure 
were  discovered  during  our  research  or  interviews,  as  the  program  currently  meets  all 
requirements. 

There  is  no  formal  training  provided  for  the  system,  as  the  website  is  fairly 

intuitive. 
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Finally,  no  remaining  redundancies  exist  within  the  hardware  of  the 
system  based  on  the  merger  between  the  East  Coast  and  West  Coast  staffs.  However, 
some  redundancies  may  be  present  within  the  personnel  management. 

b.  User  Feedback 

The  following  tables  capture  the  User  Survey  results  for  the  SURFOR 
Web.  The  quantitative  number  of  responses  received  was  adequate  based  on  the  numbers 
of  surveys  that  were  distributed  by  CNSF  N6.  The  survey  responses  received  were  from 
Active  Duty  military,  Civil  service  employees  and  Defense  contractors  from  various 
departments  within  CNSF. 
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1 .  Ease  of  access  to  system. _ 12 _ 6 _ 5 _ 1 _ 

2.  System  easy  to  use. _ 11  1 _ 5 _ 3 _ 4 _ 

3.  Have  confidence  in  system.  8  4  2  10 

4.  Degree  of  personal  control  over 

the  system. _ 2  6  4 _ 3 _ 7 _ 2 _ 

5.  System  has  up  to  date  hardware.  4  4  2 _ 2 _ 3 _ 4 _ 2 _ 1 _ 2 

6.  System  has  up  to  date  software.  5  6  1 _ 2 _ 7 _ 3 _ 

7.  Interoperable  with  other 

systems. _ 6  14 _ 1 _ 2 _ 5 _ 3 _ 1 _ 1 

8.  System’s  response  time. _ 6  6 _ 3 _ 6 _ 3 _ 

9.  Contingency  procedures  when 

system  is  down. _ 7  2  1 _ 1 _ 1 _ 2 _ 2 _ 5 _ 1 _ 2 

10.  System  responsiveness  to 

changing  user  needs. _ 5  4  1 _ 1 _ 1 _ 3 _ 6 _ 1 _ 2 

1 1 .  Flexibility  of  the  system  to 

produce  professional  reports. _ 4  4  3 _ 1 _ 4 _ 2 _ 3 _ 2 _ 1_ 

12.  Report  data  is  verifiable. _ 5  3  2 _ 2 _ 3 _ 6 _ 1 _ 2 

13.  Fast  response  time  from  support 

staff  to  remedy  technical  issues.  5  5  1 _ 1 _ 6 _ 6 _ 

14.  Positive  attitude  from  Help 

Desk  to  users. _ 5  6 _ 1 _ 7 _ 5 _ 

15.  Fast  response  time  from  Help 

Desk  staff  to  remedy  problems.  7  4 _ 1 _ 6 _ 6 _ 

16.  Ability  of  the  system  to 
improve  users’  personal 

productivity. _ 6  5  1 _ 4 _ 7 _ 1 _ 

17.  Extent  of  user  training.  264  52311 

18.  Ability  of  the  system  to 
enhance  the  learning  experience 

of  the  users’. _ 3  3  6 _ 3 _ 4 _ 4 _ 1 

19.  Documentation  to  support  user 

training  is  provided. _ 4  2  3 _ 3 _ 5 _ 1 _ 3 _ 2 _ 1_ 

20.  System  enhances  warfighter 


readiness. _ 9  3 _ 4 _ 5 _ 3 

2 1 .  System  provides  tangible 


Table  9.  CNSF  Web  User  Feedback  Numerical  Responses 
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Dissatisfied 


Importance 

Satisfaction 

Very 

Important 

Important 

Somewhat 

Important 

Not 

Important 

N/A 

Very 

Satisfied 

Satisfied 

Somewhat 

Satisfied 

Dissatisfied 

N/A 

1 .  Ease  of  access  to 
system. 

100% 

0% 

0% 

0% 

0% 

50% 

42% 

8% 

0% 

0% 

2.  System  easy  to 
use. 

92% 

8% 

0% 

0% 

0% 

42% 

25% 

33% 

0% 

0% 

3.  Have  confidence 
in  system. 

67% 

33% 

0% 

0% 

0% 

17% 

83% 

0% 

0% 

0% 

4.  Degree  of 

personal  control 
over  the  system. 

17% 

50% 

33% 

0% 

0% 

25% 

58% 

17% 

0% 

0% 

5.  System  has  up  to 
date  hardware. 

33% 

33% 

17% 

0% 

17% 

25% 

33% 

17% 

8% 

17% 

6.  System  has  up  to 
date  software. 

42% 

50% 

8% 

0% 

0% 

17% 

58% 

25% 

0% 

0% 

7.  Interoperable 
with  other 
systems. 

50% 

8% 

33% 

8% 

0% 

17% 

42% 

25% 

8% 

8% 

8.  System’s 

response  time. 

50% 

50% 

0% 

0% 

0% 

25% 

50% 

25% 

0% 

0% 

9.  Contingency 
procedures  when 
system  is  down. 

58% 

17% 

8% 

8% 

8% 

17% 

17% 

42% 

8% 

17% 

10.  System 

responsiveness  to 
changing  user 
needs. 

42% 

33% 

8% 

8% 

8% 

25% 

50% 

0% 

8% 

17% 

1 1 .  Flexibility  of  the 
system  to 
produce 
professional 
reports. 

33% 

33% 

25% 

8% 

0% 

33% 

17% 

25% 

17% 

8% 

12.  Report  data  is 
verifiable. 

42% 

25% 

17% 

0% 

17% 

25% 

50% 

8% 

0% 

17% 

13.  Fast  response 
time  from 
support  staff  to 
remedy  technical 
issues. 

42% 

42% 

8% 

0% 

8% 

50% 

50% 

0% 

0% 

0% 

14.  Positive  attitude 
from  Help  Desk 
to  users. 

42% 

50% 

0% 

0% 

8% 

58% 

42% 

0% 

0% 

0% 

15.  Fast  response 
time  from  Help 
Desk  staff  to 
remedy 
problems. 

58% 

33% 

0% 

0% 

8% 

50% 

50% 

0% 

0% 

0% 

16.  Ability  of  the 
system  to 

50% 

42% 

8% 

0% 

0% 

33% 

58% 

8% 

0% 

0% 
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improve  users’ 

personal 

productivity. 

17.  Extent  of  user 
training. 

17% 

50% 

33% 

0% 

0% 

42% 

17% 

25% 

8% 

8% 

18.  Ability  of  the 
system  to 
enhance  the 
learning 

experience  of  the 
users’. 

25% 

25% 

50% 

0% 

0% 

25% 

33% 

33% 

0% 

8% 

19.  Documentation 
to  support  user 
training  is 
provided. 

33% 

17% 

25% 

25% 

0% 

42% 

8% 

25% 

17% 

8% 

20.  System  enhances 
warfighter 
readiness. 

75% 

25% 

0% 

0% 

0% 

33% 

42% 

25% 

0% 

0% 

21.  System  provides 
tangible  financial 
benefit. 

42% 

25% 

17% 

0% 

17% 

25% 

42% 

0% 

8% 

25% 

22.  Relevant  data 
and  metrics  are 
captured  by  the 
system. 

25% 

58% 

8% 

0% 

8% 

25% 

42% 

25% 

0% 

8% 

23.  System  is  well 
aligned  across 
the  CNSF 
claimancy. 

50% 

33% 

8% 

0% 

8% 

33% 

42% 

17% 

0% 

8% 

24.  System  supports 
the  goals  of  the 
SWE. 

67% 

33% 

0% 

0% 

0% 

42% 

50% 

8% 

0% 

0% 

Table  10.  CNSF  Web  User  Feedback  Percentage  Responses 


Results  of  respondent’s  experience  and  usage: 


|  How  long  have  you  been  in  the  current  position?  (Number  of  respondents/percentage) 

Less  than  1  year 

1-3  years 

3-5  years 

More  than  5  years 

0/0% 

5  /  42% 

1/8% 

6  /  50% 

Approximately  how  many  hours  a  day  do  you  use  this  system  in  the  performance  of  your  duties?  (Number 

|  of  respondents/percentage) 

Almost  never 

<  1  hour/day 

1-2  hours/day 

3-4  hours/day 

5-6  hours/day 

>  6  hours/day 

0/0% 

0/0% 

3/25% 

4/33% 

2/17% 

3/25% 

Table  1 1 .  CNSF  Web  User  Feedback  Respondent  Data 
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Users  were  primarily  positive  in  their  responses.  For  the  entire  list  of 
negative,  positive,  and  other  comments,  see  Appendix  C. 
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V.  FINDINGS  AND  FOLLOW-ON  WORK 


Reduced  budgets  and  increased  scrutiny  of  staff  budget  obligations  forced  the 
Commander,  Naval  Surface  Forces  (CNSF)  Comptroller  to  seek  an  evaluation  of  the 
current  information  technology  investments.  The  ultimate  goal  of  any  Department  of  the 
Navy  (DoN)  information  technology  program  is  an  outcome  that  positively  contributes  to 
mission  effectiveness,  as  opposed  to  simply  striving  to  meet  pre-existing  contractual 
requirements  (Department  of  the  Navy,  2001b).  The  project  sponsor’s  evaluation  request 
will  not  only  lead  to  possible  resource  efficiencies,  but  will,  more  importantly,  highlight 
the  need  for  overarching  information  technology  management  that  supports  the 
subordinate  commands  within  the  CNSF  claimancy  and  the  DoN’s  mission  effectiveness 
goals. 

This  chapter  provides  the  results  of  evaluations  conducted  on  the  three  major 
CNSF  information  technology  programs:  Training  and  Operational  Readiness 
Information  Services  (TORIS),  Continuous  Monitoring  Program  (CMP),  and  CNSF  Web. 
It  includes  recommendations  for  follow-on  research. 

A.  PROGRAM  EVALUATIONS 

As  discussed  in  Chapter  IV,  the  project  evaluation  method  was  to  determine  the 
programs’  progress  in  meeting  cost,  schedule,  and  performance  goals  by  focusing  on  five 
core  principles:  mission,  performance,  management,  financial,  and  technical.  The 
following  evaluates  each  program  on  those  principles. 

1.  Training  and  Operational  Readiness  Information  Services  (TORIS) 

The  mission  set  of  TORIS  is  mandated  to  support  the  Surface  Force  Training 
Manual.  Based  on  feedback  received  from  the  program  manger  and  end  users,  TORIS 
meets  or  exceeds  expectations  in  terms  of  mission  support.  While  the  system  primarily 
provides  relief  from  the  administrative  burdens  associated  with  tracking  the  training 
proficiencies  for  the  respective  ships  within  the  CNSF  claimancy,  it  also  provides  a  quick 


71 


and  effective  means  for  senior  leadership  in  all  Navy  echelons  to  evaluate  fleet  readiness, 
including  the  ability  to  get  underway  while  meeting  prescribed  training  goals. 

During  the  course  of  the  research,  no  major  negative  performance-related  issues 
were  discovered.  While  this  certainly  seems  positive,  unfortunately,  we  found  no 
performance  benchmarks  by  which  to  measure  the  performance  of  the  software 
application.  This  makes  an  accurate  determination  theoretical  at  best.  The  statement  of 
work  in  the  contract  does  not  measure  the  performance  of  the  system;  instead,  it  is 
designed  to  mandate  the  requirements  for  the  contractor’s  performance.  Therefore, 
indicators  that  measure  downtime,  trouble  calls,  and  software  bugs  are  missing.  These 
are  all  factors  that  could  assist  managers  in  determining  overall  performance. 

Management  of  the  system  is  outlined  by  the  statement  of  work,  and  by  every 
account,  the  Afloat  Training  Group  (ATG)  staff  manages  the  system  well.  A 
configuration  control  board  is  in  place  that  makes  determinations  on  what  functionality 
and/or  policies  require  modification,  addition,  or  deletion  within  the  system.  By  all 
accounts,  there  is  no  oversight  of  the  TORIS  system  by  any  technical  subject  matter 
experts  within  the  CNSF  staff,  such  as  CNSF  N6. 

Concerning  system  finances,  the  TORIS  program  demonstrates  no  accurate 
measure  for  a  capital  return  on  investment.  This  may  hinder  the  Surface  Warfare 
Enterprise  (SWE)  leadership  decision  making  in  terms  of  preventing  an  accurate 
determination  of  which  programs  are  providing  the  greatest  mission  support  in  terms  of 
cost  and  return  on  investment. 

Regarding  technical  competency,  TORIS  employs  common  and  up-to-date 
programming  languages  that  are  compatible  with  other  information  systems  employed  by 
the  DoN.  During  the  research  phase  of  this  project,  it  was  found  that  no  software  or 
hardware  redundancies  exist  within  the  CNSF  claimancy.  This  is  important  to  note 
because  of  the  merger  of  the  CNSL  and  CNSP  staffs. 
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2.  Continuous  Monitoring  Program  (CMP) 

The  mission  set  of  CMP  is  directed  by  the  policies  promulgated  in  the  CNSFINST 
4440.1.  Based  on  the  authors’  personal  experience  afloat,  program  manager  feedback, 
and  user  feedback,  CMP  is  clearly  providing  an  efficient  resource  to  manage  the  myriad 
of  readiness  and  services  pulse  points  that  CNSF  and  Commander,  Submarine  Force  units 
are  required  to  maintain  for  the  highest  level  of  ship  logistics  readiness.  The  summarized 
pulse-point  report  in  the  stop-light  format  provides  all  associated  users  with  an  effective 
and  quick  means  to  ascertain  weak  areas  within  a  specific  unit,  class  of  ships,  or  the 
entire  fleet. 

CMP  system  performance  appears  to  operate  efficiently.  However,  like  TORIS, 
while  no  major  software  related  issues  were  discovered,  there  are  also  no  performance 
benchmarks  in  place  to  measure  and  track  actual  performance. 

CNSF  N411  is  responsible  for  the  management  of  CMP,  and  any  change  to  the 
system  is  done  in  accordance  with  the  policies  set  forth  by  the  CNSFINST  4440.1. 
However,  much  like  TORIS,  there  is  no  oversight  of  this  application  by  a  technical  code, 
such  as  CNSF  N6.  This  could  present  substantial  deficiencies  resulting  in  faulty  changes, 
leading  to  wasted  resources. 

With  regard  to  system  finances,  the  CMP  program  demonstrates  no  accurate 
measure  for  a  capital  return  on  investment.  Like  TORIS,  this  may  hinder  SWE 
leadership  decision  making  in  terms  of  preventing  an  accurate  determination  of  which 
programs  are  providing  the  greatest  mission  support  in  terms  of  cost  and  return  on 
investment. 

Finally,  as  to  technical  competency,  CMP  employs  common  and  up-to-date 
programming  languages  that  are  compatible  with  other  DoN  information  systems. 
During  the  research  phase  of  this  project,  it  was  found  that  no  software  or  hardware 
redundancies  exist  within  the  CNSF  claimancy.  This  is  important  to  note  because  of  the 
merger  of  the  CNSL  and  CNSP  staffs. 
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3.  Commander,  Naval  Surface  Forces  (CNSF)  Web 

CNSF  Web  was  implemented  to  provide  an  efficient  and  consolidated  means  of 
sharing  information  between  east  and  west  coast  staffs,  along  with  providing  a  means  for 
units  to  host  web  site  services  and  access  TYCOM  instructions,  policies,  directives,  and 
other  general  information.  The  program  appears  to  adequately  support  its  mission  of 
force  alignment,  e.g.,  communications  are  facilitated  between  commands  and  echelons. 

CNSF  Web  appears  to  perform  satisfactorily.  However,  like  the  other  two 
programs,  there  are  no  performance  benchmarks  in  place  to  measure  and  track  actual 
performance.  Adding  these  checks  could  enhance  user  experience,  potentially  resulting 
in  higher  usage  rates. 

During  the  research  phase  of  this  project,  day-to-day  management  of  the  system 
shifted  from  the  West  Coast  to  the  East  Coast.  Complicating  the  issue  was  the  admission 
from  the  project  sponsor  that  a  redundancy  existed  between  the  program  staffs  on  both 
coasts.  In  addition,  due  to  the  amount  of  system  administrators,  it  can  be  concluded  that 
management  of  the  system  is  extremely  challenging.  Unlike  TORIS  and  CMP,  CNSF 
Web  does  not  possess  any  written  policies  to  delineate  the  specific  responsibilities  of  the 
system  managers. 

With  respect  to  the  financial  aspect  of  the  system,  CNSF  Web  demonstrates  no 
accurate  measure  for  a  capital  return  on  investment.  Like  the  other  two  programs,  this 
may  hinder  SWE  leadership  decision  making  in  terms  of  preventing  an  accurate 
determination  of  which  programs  are  providing  the  greatest  mission  support  in  terms  of 
cost  and  return  on  investment. 

Concerning  technical  competency,  CNSF  Web  employs  common  and  up-to-date 
programming  languages  that  are  compatible  with  other  DoN  information  systems. 
During  the  research  phase  of  this  project,  it  was  found  that  no  software  or  hardware 
redundancies  exist  within  the  CNSF  claimancy.  This  is  important  to  note  because  of  the 
merger  of  the  CNSL  and  CNSP  staffs. 
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4. 


User  Feedback 


The  feedback  from  the  users  for  all  three  programs  was  grouped  into  five  mission 
areas  that  were  detailed  in  Chapter  IV.  The  mission  areas  were  used  to  develop  the 
survey.  In  summary,  more  than  87%  of  the  total  respondents  who  reported  that  these 
three  programs  mission  areas  were  either  “Important”  or  “Very  Important”  in  supporting 
the  goals  of  the  Surface  Warfare  Enterprise.  Relatively  few  respondents — less  than 
13% — reported  negative  feedback.  Chapter  IV  provides  a  detailed  breakdown  of  the 
respondents  reviews.  For  a  list  of  hand- written  comments,  refer  to  Appendices  A  through 
C. 


B.  FINDINGS  AND  RECOMMENDATIONS 

This  section  answers  each  research  question,  and  it  offers  two  types  of 
suggestions.  The  first  set  of  suggestions  deals  with  improvements  that  CNSF  can  employ 
in  the  short  term.  The  second  set  presents  options  for  follow-on  research  activities. 

1.  Research  Question  Findings 

First,  to  answer  the  specific  research  questions  delineated  in  Chapter  I: 

What  programs  are  currently  being  funded? 

The  programs  currently  being  funded  are  TORIS,  CMP  and  CNSF  Web,  and  they 
were  the  subject  of  this  report.  CNSF  expended  approximately  $5.5M  during  Fiscal  Year 
2009  on  these  programs  with  the  following  breakdown: 

•  TORIS:  $2.3M 

•  CMP:  $997K 

•  CNSF  Web:  $2.2M 

Why  are  they  being  funded,  e.g.,  what  programs  and  commands  do  they  affect? 

TORIS  supports  the  training  readiness  of  the  fleet,  building  toward  the  currently 
available  war-fighting  capability  of  the  force  in  three  ways.  First,  through  onboard 
training  record  processes,  it  provides  command  leadership  with  a  snapshot  of  current 
training  proficiency  for  all  warfare  areas,  and  it  allows  shipboard  leadership  to  develop 
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plans  to  improve  weaker  areas  and  reinforce  stronger  areas.  Second,  through  the  time- 
shortened  inspection  certification  processes,  it  facilitates  ship’s  deployment  certification 
faster.  Finally,  via  the  automatic  population  of  Defense  Readiness  Reporting  System 
(DRRS)  via  TORIS’s  Training  Figure  of  Merit  (TFOM)  metrics,  it  provides  Office  of  the 
Secretary  of  Defense  (OSD)  the  readiness  level  of  any  Navy  unit,  and  it  supports  critical 
decision  making  process  to  deploy  units  in  a  timely  manner  based  on  accurate,  up-to-date 
information. 

The  primary  mission  need  of  CMP  is  fleet  readiness,  and  the  program  supports 
this  by  bringing  attention  to  potential  supply  department  deficiencies,  which  could 
degrade  into  significant  problem  areas  for  ships  in  training  or  on  deployment.  It  does  this 
through  the  extraction  of  data,  called  pulse  points  from  the  following  Information 
Systems:  R-Supply,  Micro-Snap,  Food  Service  Management,  and  ROM  II.  In  addition,  it 
provides  95%  of  required  Supply  Figure  of  Merit  (SFOM)  data  to  Defense  Readiness 
Reporting  System-Navy  (DRSS-N). 

The  primary  mission  support  function  of  the  CNSF  Web  is  force  alignment.  It 
links  the  primary  staff  functions  of  all  echelons  within  the  chain  of  command,  from  the 
deck-plate  level  to  the  Commander,  Naval  Surface  Forces,  in  one  electronic  location.  It 
also  provides  the  resource  for  staffs  to:  publish  policies  and  directives,  follow  various 
operational  metrics,  and  communicate  with  each  other  on  pertinent  topics.  It  is  the 
primary  tool  utilized  by  CNSF  to  meet  the  Chief  of  Naval  Operations  (CNO)  FORCEnet, 
“Alignment”  and  Sea  Enterprise  goals  along  with  the  business  requirements  of  the  SWE. 
Finally,  it  appears  to  provide  an  enterprise  solution  for  the  claimancy,  as  a  certified 
Enterprise-Level  collaboration  portal. 

What  commands  do  they  support? 

All  three  programs  support  the  SWE,  all  units  within  the  CNSF  claimancy,  and 
echelons  above  the  TYCOM  level. 

What  are  the  specific  services  to  be  supported  as  identified  by  contract  terms? 

The  specific  services  are  detailed  in  the  Statement  of  Work  within  each  contract, 
which  can  be  found  in  Chapter  III. 
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Are  there  redundancies  in  services  provided  amongst  the  various  programs? 

There  appear  to  be  no  redundancies  in  hardware  or  software  systems  for  each  of 
the  three  programs  in  question.  However,  there  may  be  some  duplication  of  management 
efforts  within  CNSF  Web. 

2.  Recommendations 

In  conclusion,  the  following  are  four  specific  recommendations  for  positive 
change  that  CNSF  could  undertake  in  the  short  term,  including  two  follow-on  research 
project  concepts. 


a.  Improve  Managerial  Oversight 

Based  on  DoN  information  technology  systems  and  applications  policies, 
there  is  little  to  no  expectation  that  the  CNSF  Comptroller  is  able  to  ascertain  the  fleet 
requirement  of  any  of  the  programs  that  were  researched  in  this  project.  The  current 
funding  process  calls  for  the  applicable  program  manager  to  submit  a  program 
justification  statement  along  with  their  specific  request  for  funding.  These  justification 
documents  seemed  to  provide  little  in  terms  of  managerial  oversight.  Similarly,  they 
offer  little  means  for  the  Comptroller  to  discern  the  funding  priorities  within  the  CNSF 
claimancy. 

CNSF  N6  is  the  most  technically  equipped  to  determine  the  funding 
priorities  of  all  information  technology  applications,  and  it  should  provide  SWE  level 
management  oversight  to  ensure  that  these  programs  are  supporting  the  mission  and  goals 
set  forth  by  the  CNSF  Commander  and  SWE  Charter. 

b.  Establish  Key  Performance  Indicators 

Key  Performance  Indicators  (KPIs)  apparently  do  not  exist  for  the  three 
programs,  as  explained  in  their  performance  evaluations.  We  recommend  that  all  future 
systems  have  relevant,  clearly  defined,  and  measurable  KPIs  built-into  the  various 
programs.  Additionally,  generate  KPIs  for  all  current  systems.  KPIs  will  provide  a 
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crucial  evaluative  component  necessary  for  managing  and  adjusting  any  complex  process 
or  mechanism,  including  direct  customer/user  feedback  pinpointing  areas  for 
improvement. 


c.  Design  and  Refine  upon  Attributes  of  Successful  Information 
Technology  Systems 

As  detailed  in  Chapter  IV,  successful  information  technology  systems  can 
be  characterized  through  the  demonstration  of  six  quality  attributes.  Mastery  of  those 
attributes — usability,  interoperability,  responsiveness,  maintainability,  scalability,  and 
standardization — can  improve  customers’  experience,  ease  the  path  for  future 
modifications,  and  increase  each  program’s  mission  supportability.  While  easiest  to 
build  upon  in  the  system  design  phase,  program  managers  and  CNSF  technical  experts 
can  continually  diagnose  and  refine  their  systems  for  improved  performance. 

d.  Follow-up  this  Baseline  Assessment  in  the  Future 

A  baseline  assessment  is  designed  to  establish  the  snapshot  picture  of  the 
status  quo.  We  recommend  that  CNSF  conduct  periodic  follow-on  assessments  to  ensure 
that  priorities  are  understood  and  met.  Failure  to  do  so  should  not  be  an  option. 
Therefore,  as  described  in  Chapter  II,  subsequent  assessment  should  follow  up  this 
baseline  assessment  in  order  to  maximize  the  chance  of  the  project  sponsor  reaching  its 
goals. 


e.  Conduct  Follow-on  Research  to  Develop  an  Information 
Technology  Strategy  and  Acquisition  Strategy 

This  project  begets  two  follow-on  research  projects.  The  next  group 
should  develop  an  information  technology  strategy,  which  would  be  a  top-down 
expression  of  the  command’s  determination  of  its  information  technology  objectives. 
The  third  and  final  group  could  compose  an  acquisition  plan  as  a  way-ahead  for  CNSF 
resource  allocation. 
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Developed  strategies  follow  the  DoN  Information  Technology  Capital 
Planning  Guide  because  it  defines  each  strategy  and  details  the  recommended  steps  to 
establish  them. 


Particularly,  it  calls  for  the  Information  Technology  strategy  to: 

State  senior  leadership  priorities  for  IM/IT;  reflect  vertically,  the  overall 
non-IM/IT  mission  priorities  of  DoN  and  DoD,  and  provide  a  focused 
framework  for  linked  implementation  of  all  IM/IT  initiatives  within  DoN; 
establish  performance  measures  to  determine  progress  towards 
accomplishing  objectives.  (Department  of  the  Navy,  2001b) 

The  acquisition  strategy  could  be  coordinated  with  the  information  technology 
strategy.  The  way  ahead  will  likely  include  continual  diagnosis  of  the  ongoing 
procurement  process  and  recommended  actions  for  CNSF  resource  allocation. 
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APPENDIX  A:  TORIS  USER  FEEDBACK 


Written  feedback  from  user  surveys,  negative  comments  regarding  this  system 

first: 


•  “System  did  not  convert  old  program  (ship)  file  to  new  (TORIS) 
update.” 

•  “TORIS  system  may  need  to  be  more  fluid  in  change  to  instruction 
changes  and  new  equipment  implemented  to  the  fleet.” 

•  “Some  improvement  in  training  (user  friendly)  may  be  required.” 

•  “Systems  slow  on  inputs  for  CCR.” 

•  “Updating  cards  not  being  updated  have  to  log-off  and  login  about 
two  times.” 

•  “Never  received  any  system  training  other  than  that  provided  by 
divisional  personnel.” 

•  Seems  to  be  very  difficult  to  query  the  database  for  specific 
information  and  seems  to  require  an  official  request  for  what 
should  be  fairly  simple  database  function.” 

•  “N/A  function  should  be  ATG/TRAINO  only  function.” 

•  “Deletion  of  cards  past  seven  days  requires  system  admin  access.” 

•  “Ship  inputted  data  vs.  ATG  inputted  data  is  not  clearly  identified. 
Recommend  to  restrict  ships  ability  to  input  data  cards  when  not 
self-access.” 

•  “Not  able  to  see  ship  trends  through  a  database  query.” 

•  “Using  TORIS  takes  a  lot  of  time.  The  system  is  slow.” 

•  “TORIS  is  not  specific  in  capturing  actual  operator  proficiency 
over  procedural  proficiency.” 

•  “TORIS  log  in  is  problematic.  The  system  continually  crashes.” 

•  “The  new  DDGs  on  the  waterfront  have  different  equipment  that 
does  not  relate  completely  to  some  of  the  TORIS  Cards 
requirements,  i.e  DMRs  have  replaced  WSC-3s,  classified  and 
unclassified  servers  no  longer  have  tape  drives  and  cannot  store 
backups  in  a  remote,  secure  location.” 

•  “MSP  doesn’t  always  work,  freezes  and  doesn’t  always  update.” 

•  “The  tough  books  work  fine,  but  we  need  more  of  them.” 
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“There  is  little  or  no  training.  I  received  my  training  on  the  job 
and  by  trial  and  error.” 

“Inaccurate  at  times.” 

“Difficult  to  learn,  not  intuitive.” 

“Easy  to  falsify  data.” 

“System  not  able  to  easily  tailored  to  specific  platforms  by  end 
users.” 

“System  does  not  appear  to  be  stable.  Changes  often  are  not  saved 
and  system  “locks-up.” 

“Transferring  data  from  the  laptop  while  on  the  ship  to  the  LAN.” 

“I  can’t  use  my  CAC  access  to  log  on.” 

“I  have  to  change  my  password  too  often.” 

“Sometimes  the  system  is  too  slow  to  load.” 

“Initial  and  upgrade  user  training  is  generic.  Recommend 
including  hands-on  training  sessions  with  actual  fleet  scenarios  to 
increase  knowledge  base  and  detect  program  flaws/bugs.  This 
would  allow  for  direct  feedback  from  end-users  to  enhance  the 
implementation  of  newer  versions.” 

“Stand  alone  TORIS  program  support/help  link  could  be  more 
user-friendly.  This  tool  is  very  important  when  training  teams  are 
onboard  ships  underway  and  unable  to  directly  interact  with 
technical  support  personnel.” 

“MSP  Trainer  Time  report  sub-events  section  includes  a  column 
for  tracking  time  spent  conducting  Full  Power  Demo.  This  column 
is  never  used  during  training  events  and  should  be  changed  to 
classroom  training  to  more  accurately  capture  time  spent  during 
LTTs.  Additionally,  MSP  data  and  TORIS  data  are  not  interlinked 
for  information  sharing  purposes.  A  link  could  be  added  to  the 
TORIS  database  that  allows  the  user  to  view  the  number  of  LTTs 
and  ULTRA  events  conducted  per  ship  for  a  selected  period.” 

“In  reference  to  TORIS  Supply,  if  a  call  is  checked  as  N/A  a 
trouble  call  needs  to  be  put  in  to  get  it  changed.  Something  as 
simple  as  a  check  in  the  box  should  be  the  users  call  to  check  or 
uncheck  without  a  trouble  call.  Or  editing  the  N/A  call  without 
assistance.  Additionally,  it  takes  administrator  longer  to  provide 
assistance  with  a  trouble  call  which  delays  the  TORIS  posting 
process.  Thje  problem  is  as  follows:  R-Supply  (Unit  Level) 
Accountability,  RSA-003 — LOCATION  AUDIT,  the  table  for 
DLR  and  REPAIR  PARTS  do  not  calculate  properly  when  posting; 
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and,  the  numbers  inputted  in  #INVENTORIED,  #CORRECT,  and 
ACCURACY  blocks  do  not  post/record  while  posting  and  do  not 
show  in  the  Summary  Report  when  viewing  the  report  for  printing. 
There  has  been  3  trouble  call  attempts  to  correct  this  issue.” 

•  “In  reference  to  TORIS  MSP,  users  are  unable  to  make  changes  to 
correct  time  onboard  for  training  and  posting  trainer  times.” 

Positive  comments  about  this  system: 

•  “The  help  desk  is  very  expedient  in  fixing  problems.” 

•  “Transfer  from  Team  Lead  to  TLO  information  is  good.” 

•  “Tough  book  is  durable,  small,  and  easy  to  lock  away.” 

•  “System  provides  current  status  (when  accurate).” 

•  “System  use  appears  to  be  fairly  simple.” 

•  “If  used  correctly,  allows  for  planning.” 

•  “If  used  properly,  allows  for  ships  to  see  mission  area  readiness.” 

•  “Collective  based  data  points  allow  for  multi-scenario  gain  of 
CDPs.” 

•  “The  system  is  available  for  the  entire  chain  of  command  to  view  it 
and  understand.” 

•  “The  system  is  organized.” 

•  “It’s  better  than  nothing.” 

•  “The  updates  to  TORIS  have  been  swift  and  without  any  major 
interruptions.” 

•  “It  is  fairly  easy  to  use  basic  functions.” 

•  “Ease  of  tracking  events.” 

•  “Quantitatively  demonstrates  readiness.” 

•  “Gives  direct  feedback  on  areas  that  need  to  be  trained  to.” 

•  “Provides  quantifiable  status  of  ship’s  training  to  ISIC.” 

•  “Pretty  easy  interface  to  operate.” 

•  “Nothing  truly  positive.  It’s  a  mandatory  program  for  shipboard 
training.” 

•  “Easy  to  look  up  information.” 

•  “Easy  to  input  information.” 
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•  “Easy  to  access  specific  information.” 

•  “Fairly  easy  to  use.” 

•  “Provides  clear  data  on  where  a  ship  stands  during  the  training 
cycle.” 

•  “Data  cards  are  easy  to  access  and  update  and  accurately  populate 
the  CCR  tracker  ensuring  data  reflects  current  proficiency.” 

•  “Data  files  are  easy  to  generate  and  export  to  ship  TORIS 
database.” 

•  “Help  desk  and  TORIS  support  personnel  are  very  knowledgeable 
and  helpful  when  resolving  issues.  Tough  books  are  updated 
regularly  to  ensure  dependability  and  that  newest  software  versions 
are  available.” 

•  “Everything  is  on  one  site.  Big  plus  you  don’t  have  to  have 
multiple  sites  to  search  through  to  get  what  you  need.” 

•  “CTRs  are  on  top  of  the  trouble  calls  never  an  issue  getting  things 
updated  or  fixed.” 

•  “Access  is  simple,  easy  to  follow  directions  and  maneuvering 
through  TORIS  is  quick  and  painless  for  the  most  part.” 


Other  amplifying  information: 

•  “With  more  official  training,  the  time  required  for  personnel  to 
become  proficient  in  its  utilization  could  be  maximized.” 

•  “Ships  are  not  using  TORIS  properly.  Establish  a  level  of 
knowledge  representative  for  each  mission  area  on  ships  and 
restrict  who  is  able  to  input  data.” 

•  “The  change  process  takes  way  too  long.  If  things  are  wrong,  we 
need  to  fix  the  problem,  not  wait  until  the  next  revision  to  fix  what 
is  broke.” 
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APPENDIX  B:  CMP  USER  FEEDBACK 


Written  feedback  from  user  surveys,  negative  comments  regarding  this  system 

first: 

•  “When  out  putting  files  into  Excel,  "E"  series  requisitions  tend  to 
get  messed  up.  It  is  read  as  a  formula  and  not  a  text  block.” 

•  “Inflexibility  to  combine  several  reports  into  one  package.” 

•  “In  some  cases  the  reports  only  give  the  surface  information. 
When  digging  into  the  weeds  we  find  that  GREEN  isn't  always 
GREEN.” 

•  “Program  does  not  send  or  raise  flags  when  monthly/weekly  ship's 
comments  are  not  provided  for  areas  identified  by  being  RED  or 
YELLOW  by  the  users  submitting  the  data.  Examples,  areas  that 
are  flagged  red  should  force  the  users  to  provide  comments 
regardless  of  whether  it's  a  system  problem  or  user  self-inflicted.” 

Positive  comments  about  this  system: 

•  “Program  is  constantly  evolving.  That's  a  good  thing.” 

Other  amplifying  information: 

•  “May  want  to  offer  some  training  on  generating  self-defining 
reports.” 
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APPENDIX  C:  CNSF  WEB  USER  FEEDBACK 


Written  feedback  from  user  surveys,  negative  comments  regarding  this  system 

first: 


•  “SIPRNet  version  is  slow,  clunky  and  is  not  user  friendly.” 

•  “Front  page  seems  to  never  change  even  though  I  know  new 
content  is  being  added  all  the  time.” 

•  “Main  site  seems  disorganized  and  search  is  often  broken  making 
it  hard  to  find  documents  I'm  looking  for  -  CLASSRON  Web  is 
organized  much  better.” 

•  “Being  a  member  of  the  SURFORWeb  Team,  my  biggest  concern 
is  the  size  of  my  team  being  able  to  keep  up  with  the  number  of 
requests  that  members  submit  to  us  and  how  quickly  we  are  able  to 
satisfy  their  needs.  Therefore;  Time  constraints  and  resources  are 
my  major  concern  with  the  system.” 

•  “Turnover.  In  two  respects.  When  someone  leaves  a  position  and 
they  have  a  'good'  track  record  of  letting  the  system  work  for  them, 
document  management,  list  item  management,  expiring 
announcements  /  content,  it  often  does  not  convey  well  to  the 
incoming  person  and  they  let  a  good  process  die.” 

•  “Naysayers  /  non-believers  /  stove-pipers.” 

•  “Inability  to  generate  reports  in  a  clean  format  ready  for 
presentation  to  Admiral  Staff.  Looking  for  the  ability  to  generate 
an  automated  report  from  a  list  without  first  exporting  it  to  Excel.” 

•  “System  timeouts.  I  will  modify  a  page  or  add  new  information 
and  receive  a  blank  white  page  lacking  any  error  message.” 

•  “Changing  permissions  of  a  child  site  back  to  the  parent  site  once 
custom  permissions  have  been  created.” 

•  “Maintenance  is  scheduled  during  East  coast  work  hours.” 

•  “Navigation  is  not  intuitive  for  every  function.” 

•  “Access  can  be  difficult,  sometimes  the  system  does  not  respond  to 
the  normal  means  of  access,  requiring  operator  to  back  out  and 
start  again.  System  requires  both  CAC  and  User/Name  Password 
for  access.  Worse.  Every  file  download  requires  user 
name/password  access  -  not  productive.” 

•  “What  kind  of  organization  in  today's  Navy  does  not  include  their 
senior  enlisted,  not  in  a  link,  not  even  on  an  organization 
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chart... CSO  is  listed  for  ships  and  other  commands  -  but  not  the 
senior  enlisted  -  it  is  no  wonder  they  are  having  so  many  problems 
in  the  surface  community.  The  Force  Master  Chief  in  one  link  is 
two  FMCs  ago  (Hakim  Diaz  was  replaced  by  Jerry  Hauter  who  is 
now  also  retired  and  replaced.  This  is  2009  not  1809.” 

•  “We  do  not  always  get  advised  of  outages  that  affect  web 
services.” 

•  “Software  for  command  website  development  is  Share  Point 
restricted.” 

•  ’’SharePoint  has  security  issue  with  Word  documents.  Requires 
another  User  Log-in.” 

•  “When  the  system  is  down  or  reduced  in  functionality  (which  is 
not  frequent),  the  lag  time  in  getting  things  back  up  can  sometimes 
be  considerable  since  it  is  located  on  the  opposite  side  of  the 
country.  I'm  not  sure  if  that's  the  only  factor,  but  I  know  it 
contributes.” 

•  “I'd  like  to  see  a  greater  variety  of  user  modules  (web  parts) 
available  that  better  leverage  the  large  SharePoint  development 
community.” 

•  “It  would  be  good  to  have  some  more  advanced  area  administrator- 
level  user  training  that  gets  into  the  details  of  how  to  construct 
complex  web  parts,  leverage  SharePoint  lists  and  databases  etc., 
that  cater  to  users  that  are  more  familiar  with  the  system  and  would 
like  to  use  more  complex  functionality.” 

Positive  comments  about  this  system: 

•  “Easy  access.” 

•  “Good  source  of  helpful  information.” 

•  “It's  easy  to  get  and  maintain  my  user  account  and  access  to  pages 
I  need  access  to.” 

•  ’’There's  lots  of  great  information  in  the  site.” 

•  “The  portions  of  the  site  that  have  fresh  content  and  intuitive 
look/feel  (CLASSRON  WEb  N7  pages)  are  a  pleasure  to  use.” 

•  “SURFORWeb,  designed  in  SharePoint,  provides  the  ability  to 
collaborate  on  policy,  not  only  spanning  the  distance  between  the 
two  coasts,  but  also  supports  world-wide  collaboration.” 

•  “Alerts  are  handy  features  that  help  people  answer  immediate 
questions.” 
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•  “SharePoint  is  a  versatile  tool.  It  can  help  people  manage  in  a 
variety  of  ways,  meeting  sites,  team  sites,  announcements,  lists.  It's 
just  a  great  tool!!” 

•  “Lists  capability.  Makes  managing  action  items  and  information 
very  easy  and  accessible.” 

•  “Auto  alerts  -  helps  keep  track  of  site  use  and  modifications.” 

•  “Having  the  choice  to  modify  web  parts  in  Rich  Text  Editor  and 
Source  Editor.  Makes  it  easier  to  teach  to  other  administrators  who 
may  not  have  a  code  background.” 

•  “Easy  to  navigate.” 

•  “Generally  well  laid-out,  with  respect  to  info  available.” 

•  “Wide  assortment  of  information,  has  great  potential  that  is  yet 
untapped.” 

•  “Good  place  to  park  current  info  for  use  by  others,  as  long  as  it  is 
kept  up  to  date.” 

•  “The  helpfulness  and  resourcefulness  of  the  SURFOR  Web 
services  when  approached  with  an  issue.” 

•  “I  believe  the  service  to  the  fleet  provided  by  SURFOR  web 
services  is  second  to  none.” 

•  “Shellie  Underwood  and  John  Dyar  have  been  great  to  work  with 
for  the  past  3-4  years.  We  have  had  many  challenges  with  BRAC, 
and  consolidation  of  resources.” 

•  “Provides  good  storage  capability.” 

•  “Provides  ability  to  load  Word,  Excel,  and  Power  Point 
documents.” 

•  “The  system  is  very  reliable,  works  consistently  and  is  well- 
supported  by  a  knowledgeable  local  staff. 

•  “It  has  proven  to  be  an  invaluable  tool  for  for  organizing  the 
countless  working  groups,  sub-teams  and  initiatives  that  are 
constantly  being  created.” 

•  “The  development  staff  is  unusually  helpful  and  approachable.” 

Other  amplifying  information: 

•  “Our  biggest  hurdle  is  Top  down  guidance  /  support.  Personnel 
simply  will  not  use  a  system  that  isn't  fully  supported  by  the  Front 
Office  staff.  The  Front  Office,  neither  the  VADM  nor  his  staff 
realize  the  significance  that  this  system  could  save  them  in  terms  if 
it  were  embraced.” 
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•  “Need  a  help/how  to  use  section,  recommend  another  survey  - 
from  someone  -  just  like  this  in  6  months  to  the  same  respondents.” 

•  “The  system  is  flexible,  the  support  &  development  staff 
competent  and  responsive.  I'd  like  to  see  them  have  a  greater 
degree  of  control  over  system  configuration  and/or  administration 
to  be  able  to  both  react  to  unique  user  needs  as  well  as  more 
rapidly  support  system  errors  or  downtime.  I  realize  that  much  of 
this  is  out  of  their  hands,  but  the  system  has  become  (especially 
over  the  last  1 .5  years)  critical  to  our  enterprise  goals  as  a  central 
communications  hub.” 
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